10000年1月1日 星期六

Menu



Loading menu...

2025年12月18日 星期四

Making Jetson Nano Filesystem Tolerant to Power Loss with a Read-Only Rootfs


Jetson Nano boots and runs its root filesystem directly from an SD card, which makes the system fragile under real-world power conditions. Sudden power loss can corrupt the filesystem, and in many cases the entire SD card becomes unbootable and must be fully re-imaged. After this happens a few times, the card itself may no longer be reliable at all.

This post records a set of system-level skills used to make Jetson Nano more tolerant to power cuts and unstable power sources. The focus is simple: minimize runtime writes to the SD card, reduce the chance of filesystem corruption, and—when appropriate—avoid writing to the SD card altogether.
The approaches covered include : (⭐represents the level of power-loss tolerance)

  • SWAP tuning for system stability : reducing swap-induced SD card writes ⭐⭐
  • Journald to RAM : keeping system logs off the SD card ⭐⭐⭐⭐
  • TMPFS for /tmp : avoiding temporary file writes to disk ⭐⭐⭐
  • Disable access-time logging : reducing filesystem metadata writes on read ⭐
  • Dirty page flush tuning : adjusting kernel write-back timing ⭐⭐⭐⭐
  • Enable hardware watchdog auto-reboot : automatic recovery from hard freezes ⭐
  • Root filesystem overlay via initramfs switch : eliminating runtime writes to the system partition ⭐⭐⭐⭐⭐

※  The last one, overlay mode is the ultimate protection against SD card corruption. However, it is not suitable during active development, where a writable root filesystem is often required. Running in writable mode restores flexibility, but also removes this layer of protection.

Not all methods need to be applied together. For development systems, applying ⭐⭐⭐ and above items is usually sufficient (overlay excluded).

一. SWAP tuning for system stability ⭐⭐


Swap allows the kernel to move inactive memory pages from RAM to secondary storage when RAM is under pressure. On Jetson Nano, swap typically resides on the SD card, which effectively turns the SD card into very slow and wear-prone RAM.

The swappiness parameter controls how aggressively the kernel starts using swap, expressed as a percentage threshold. The default value is 60, meaning the kernel will begin swapping relatively early even when a significant amount of RAM is still available. Lowering this value reduces SD card writes and improves tolerance to abrupt power loss, at the cost of potential lag when RAM is fully exhausted.

※ This tuning is useful in normal (non-overlay) systems. It is not needed when running with a fully read-only root filesystem and RAM overlay.

Check current swappiness:

cat /proc/sys/vm/swappiness

(Default is 60 unless previously modified.)

Set swappiness to 10 via sysctl: 

sudo vi /etc/sysctl.d/99-swappiness.conf

Add the line:

vm.swappiness=10

Apply and verify:

sudo sysctl -p /etc/sysctl.d/99-swappiness.conf
cat /proc/sys/vm/swappiness

Or simply reboot:

sudo reboot

Inspect current swap usage:

cat /proc/swaps
swapon --show
free -h

(Optional) Recreate swap file

Only needed if your current /swapfile is too small or has been removed.

Disable and remove the old swap file:

sudo swapoff /swapfile
sudo rm /swapfile

Create a new 1 GB swap file:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Enable it at boot:

sudo vi /etc/fstab

Add the line:

/swapfile swap swap defaults 0 0

Confirm:

swapon --show


二. Journald to RAM ⭐⭐⭐⭐

By default, systemd-journald can store logs on disk under /var/log/journal, which creates continuous writes on an SD card. Switching journald to volatile mode keeps logs in RAM (/run/log/journal), so they disappear after reboot and no longer wear the SD card during normal operation.

※ Not needed in full read-only + RAM overlay mode.

Switch journald to RAM-only by modify the file journald.conf:

# sudo vi /etc/systemd/journald.conf

# Change or add journal storage setting
Storage=volatile       # Always store logs in RAM (/run/log/journal)
#Storage=auto          # Use RAM if /var/log/journal does not exist, otherwise use disk
#Storage=persistent    # Always store logs on disk (/var/log/journal)

SystemMaxFileSize=32M  # Limit individual journal file size

Restart journald or reboot:

# Restart journald service
sudo systemctl restart systemd-journald

# Or reboot to apply
sudo reboot

Verify usage and location:

# Check disk usage of journal
journalctl --disk-usage

# Check actual journal storage location
journalctl --verify

# → Shows /run/log/journal if stored in RAM (volatile mode)
# → Shows /var/log/journal if stored on disk (persistent mode)

三. TMPFS for /tmp ⭐⭐⭐

/tmp is used for short-lived temporary files. Mounting it as tmpfs keeps these writes in RAM instead of on the SD card, reducing wear at the cost of some RAM usage.

※ Not needed if a read-only root filesystem with overlay is already enabled, but useful in normal writable mode.

Configure /tmp as tmpfs

# Mount /tmp in RAM (512MB)
# sudo vi /etc/fstab

# Add line:
/tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,size=512M,mode=1777 0 0

Apply without reboot (or reboot)

# Mount now without reboot
sudo mount -a
mount | grep /tmp

# Or reboot to apply
sudo reboot

Confirm usage

# Confirm /tmp is mounted as tmpfs
findmnt /tmp

# Show how much space is used in /tmp
sudo du -sh /tmp

四. Disable access-time logging on root filesystem ⭐

By default, every file or directory read can update its access time (atime) on disk. Disabling access-time logging stops these updates on read operations, while file modification times are still recorded as usual.

※ Useful in normal writable mode to reduce unnecessary SD card writes. Has no effect when a read-only overlay is active.

Edit root filesystem mount options

# Remove access-time logging on root filesystem
# sudo vi /etc/fstab

# Replace:
/dev/root / ext4 defaults 0 1

# With:
/dev/root / ext4 defaults,noatime,nodiratime,errors=remount-ro 0 1

Reboot and verify

# Reboot to apply
sudo reboot

# Output should show "noatime,nodiratime" in mount options
mount | grep 'on / '

五. Dirty page flush tuning ⭐⭐⭐⭐

When an application writes data, the kernel does not immediately write it to disk. Instead, the data is temporarily kept in RAM as dirty pages and written back to disk later in batches. This improves performance, but increases the risk of data loss if power is suddenly cut.

By flushing dirty pages to disk earlier and more frequently, the time window during which written data exists only in RAM is reduced, improving resilience against power loss.

※ Useful in normal writable mode when minimizing data loss on power failure. Has no effect in read-only or overlay modes.

Create a dirty-page tuning config

sudo vi /etc/sysctl.d/99-dirty-flush.conf

Add:

# Start background writeback at 5% dirty memory (default: 10%)
vm.dirty_background_ratio=5

# Block new writes at 10% dirty memory (default: 20%)
vm.dirty_ratio=10

# Force writeback after 5 seconds (default: 30 seconds)
vm.dirty_expire_centisecs=500

# Check for dirty pages every 1 second (default: 5 seconds)
vm.dirty_writeback_centisecs=100

Reboot and verify

sudo reboot

# Verify dirty page parameters
sysctl -a | grep dirty

六. Enable hardware watchdog auto-reboot ⭐

If the system hard-freezes, a hardware watchdog can automatically reboot the device when it stops responding. Jetson Nano provides a built-in watchdog that can be managed by systemd.

In practice, this mainly acts as a last-resort recovery mechanism for complete system hangs, and does not prevent data corruption by itself.

※ Optional. Mostly relevant for unattended or appliance-style systems; limited benefit during active development.

Check watchdog support

zcat /proc/config.gz | grep -Ei 'dog|tegra_wdt'

You should see entries indicating that the Tegra watchdog is enabled in the kernel.

Test the watchdog directly (forces reboot)

sudo sh -c 'echo V > /dev/watchdog'

If working, the system should reboot after roughly one minute.
Do NOT run this on a system with unsaved work.

Configure systemd to use the watchdog

sudo vi /etc/systemd/system.conf

Set:

RuntimeWatchdogSec=30

Reboot and confirm

sudo reboot

# Verify watchdog timeout
systemctl show --property=RuntimeWatchdogUSec

The reported value should correspond to 30 seconds.


七. Root filesystem overlay via initramfs switch ⭐⭐⭐⭐⭐

All previous sections aim to make a writable, SD-card–based system less fragile. However, for real deployments where power loss is expected, the only robust approach is to stop writing to the system partition entirely.

An initramfs-based root filesystem overlay achieves this by mounting the base root filesystem as read-only and redirecting all runtime writes to a RAM-backed overlay. The system runs normally, but any changes made at runtime are discarded on reboot unless explicitly committed in writable mode.

※ Ideal for deployment, but largely unusable during active development, where a writable root filesystem is required.
※ Any data that must persist at runtime must be stored on a separate, non-system partition; this typically requires explicit disk partitioning and mount configuration.

Clone the overlay setup tool

git clone https://github.com/lehni/root-ro.git
cd root-ro

Install the Jetson Nano preset

sudo ./install-nano.sh

This installs a custom initramfs and helper tools that allow booting into either:

  • Read-only root filesystem with a tmpfs overlay (normal safe mode)

  • Fully writable root filesystem (maintenance / upgrade mode)

Reboot into readonly or writable mode

sudo reboot-ro   # Boot with overlay (rootfs effectively read-only)
sudo reboot-rw   # Boot with normal writable rootfs

Switch mode temporarily without full reboot (for testing)

mount-ro   # Remount with overlay (read-only)
mount-rw   # Remount writable (with chroot)

The exact behavior of these commands depends on the root-ro tool’s implementation. The intended workflow is to perform development, debugging, and system updates in writable mode, then return to read-only mode for normal operation.



2025年5月14日 星期三

Ensure SD Card Backups Are Flashable to Any Same-Size Card for Jetson Nano

 ※if you are not interested in the manual steps as below, there is a script to deal with the equivalent procedures for 64GB. It might also work for the other sizes if you adjust the variable LABELED_GB to match your SD card labeled capacity in GB, but this has not been tested. 
# Make the script executable (only needed once)
# chmod +x backup_jetson_nano_64GB_img.sh
sudo ./backup_jetson_nano_64GB_img.sh jetson.img /dev/sdX
# Run without arguments to show help
# ./backup_jetson_nano_64GB_img.sh
Morerover, it may be necesary to read the last session 補說 to know how to use the fully SD card capacity before flashing.

Cloning a Jetson Nano SD card with dd creates a full-size image — but not all SD cards labeled with the same size have the identical usable capacity. Some may be a few megabytes smaller, which will lead to the flashing error : Target device is too small. 

On the other hand,  if you flash the image to the larger SD card(e.g. from 64 GB to 128 GB), The system is likely to fail to boot.

This guide shows how to safely trim the unused space at the end of the imae, align it properly, and fix GPT headers — so your .img backup can be reliably flashed to other SD cards of the same advertised size (e.g. 64GB) using Etcher or dd.

零. Full backup SD card to .img

Insert the SD card and identify the device name:
lsblk -dpno NAME,SIZE,MODEL | grep -Ev 'loop|nvme|sda|boot'
Example output:
/dev/sdb     58.2G   SD/MMC

In this case, the SD card is /dev/sdb.

Create a full image backup:

sudo dd if=/dev/sdx of=jetson_backup.img bs=16M status=progress conv=fsync

Replace /dev/sdX with your SD card device. You can rename jetson_backup.img as needed.


壹. Shrink main partition and truncate the image size

Attach image as loop device:
sudo losetup -Pf --show jetson_backup.img
Example output:
/dev/loop47

Use GParted to shrink the main partition:
sudo gparted /dev/loop47

To shrink the largest ext4 partition (usually labeled "APP" — typically the root partition in Jetson Nano images), follow the steps below:

  • Right-click on the APP partition (such as /dev/loop47p1, ext4, 59.51 GiB in the screenshot).

  • Select "Resize/Move" from the context menu.

  • In the "Resize/Move" dialog:

    • Set "Free space following (MiB)" to 8192 (i.e., 8 GiB).

    • Do NOT change "Free space preceding(MiB)" value.

    • Click the "Resize/Move" button to return to the main GParted window and confirm the change.

  • Go to Edit → Apply All Operations to apply the shrink operation.

Detach the loop device:
sudo losetup -d /dev/loop47



貳. Extend image by 1 MB buffer and align to 512 bytes

Find the partition end (in bytes):
sudo parted jetson_backup.img unit B print
Example output:
Number  Start       End            Size            File system  Name
1       14680064B   63892881407B   63878201344B    ext4         APP
In this example, the end is 63892881407.

Then:
# Add 1MB buffer (1,048,576 bytes)
# 63892881407 + 1048576 = 63893929983
sudo truncate -s 63893929983 jetson_backup.img

Align to next 512-byte boundary (required for GPT):
sudo truncate -s $(( ( $(stat -c %s jetson_backup.img) / 512 + 1 ) * 512 )) \
jetson_backup.img 



參. Fix GPT backup header and verify partition table

Start gdisk:
sudo gdisk jetson_backup.img
In the gdisk prompt, type:
w
y

Inspect the partition table to ensure there are no errors
sudo gdisk -l jetson_backup.img
sudo fdisk -l jetson_backup.img 


肆. (Optional) Compress image 

Zip the image file:
zip -9 -v jetson_backup.img.zip jetson_backup.img

Then, remove the image:
rm jetson_backup.img


補說. Flash the image to the SD card and fully use its capacity.

You can flash either the .img or .img.zip file using:

Etcher (balenaEtcher)
Or via terminal for flashing the .img:
sudo dd if=jetson_backup.img of=/dev/sdX bs=16M status=progress conv=fsync

Optional but recommended: After flashing, you may run the following to relocate the GPT backup header to the actual end of the card (avoids partition warnings):
sudo sgdisk -e /dev/sdX # Relocate the GPT backup header

Optional: Extend the ext4 partition ("APP") to use the full capacity of the SD card.
sudo growpart  /dev/sdX 1     # Expand the APP partition
sudo resize2fs -f /dev/sdX1   # Resize the ext4 filesystem
sudo gdisk -l /dev/sdX        # Verify that GPT looks correct


2023年3月31日 星期五

Implementation of STM32 as an USB Three-Classes Composite Device, HID + CDC + MSC

 

    ※ The full source code is here.

    There is some posts discussion about how to implementation an USB composite device on stm32, like this, this, this and this. But I could not find out a post to illustrate how to deal with a 3-classese composite device. After my study, it is more tricky to implement a 3-classe device than 2 classes, and those tricks do not been mentioned on the posts. This post is to record those tricks, to help the people want to implement an USB composite device on stm32.

   Here I use custom Human Interface Device(HID) + Communication Device Class(CDC, also call com port) + Mass Storage Class(MSC)  as the example, which is the single most common use-case for a 3 classes USB compsoite device.

零. Check the max endpoint number of your STM32.

    I use stm32F103 in this example, and stm32F103 supports with USB full speed mode only. 

    For Custom HID + CDC + MSC device, the amount of the endpoints are : 1 USB controling (called EP0),  2 for CDC(1 control + 1 data), 1 for MSC, and 1 for custom HID, totally those are five endpoints. 

    But Not all stm32 meet this requirement on the full speed mode: for example, STM32F2XX and STM32F4XX on full speed mode, there are only 4 endpoints. So, Here I use stm32F103, which support 8 endpoins.


壹. Use STM32CubeIDE to generate the USB classes code, and make sure they can work individually.

         As you know, while the ioc setting changed, would delete the previous generated code. So it is necessary to backup the generated USB class code then copy back.

   

    甲.  Bring CustomHID working

       On the ico for USB Device customHID setting,  we set USBD_CUSTOMHID_OUTREPORT_BUF_SIZE as 32.   

     After the code generated, we modify  CUSTOM_HID_ReportDesc_FS, which locates at the File USB_DEVICE\App\usbd_custom_hid_if.c , as :

:
/** Usb HID report descriptor. */
__ALIGN_BEGIN static uint8_t CUSTOM_HID_ReportDesc_FS[USBD_CUSTOM_HID_REPORT_DESC_SIZE] __ALIGN_END =
{

  /* USER CODE BEGIN 0 */
	0x05, 0xFF, // USAGE_PAGE(User define)
	0x09, 0xFF, // USAGE(User define)
	0xa1, 0x01, // COLLECTION (Application)
	/* 6 Bytes*/

	// The Input report
	0x05, 0x01, // USAGE_PAGE(1)
	0x19, 0x00, // USAGE_MINIMUM(0)
	0x29, 0xFF, // USAGE_MAXIMUM(255)
	0x15, 0x00, // LOGICAL_MINIMUM (0)
	0x25, 0xFF, // LOGICAL_MAXIMUM (255)
	0x75, 0x08, // REPORT_SIZE (8)
	0x95, USBD_CUSTOMHID_OUTREPORT_BUF_SIZE, // REPORT_COUNT (USBD_CUSTOMHID_OUTREPORT_BUF_SIZE)
	0x81, 0x02, // INPUT (Data,Var,Abs)

	/* 22 Bytes */

	// The Output report
	0x05, 0x02, // USAGE_PAGE(2)
	0x19, 0x00, // USAGE_MINIMUM (0)
	0x29, 0xFF, // USAGE_MAXIMUM (255)
	0x15, 0x00, // LOGICAL_MINIMUM (0)
	0x25, 0xFF, // LOGICAL_MAXIMUM (255)
	0x75, 0x08, // REPORT_SIZE (8)
	0x95, USBD_CUSTOMHID_OUTREPORT_BUF_SIZE, // REPORT_COUNT (USBD_CUSTOMHID_OUTREPORT_BUF_SIZE)
	0x91, 0x02, // OUTPUT (Data,Var,Abs)
	/* 38 Bytes */

  /* USER CODE END 0 */
  0xC0    /*     END_COLLECTION	             */

};

/* USER CODE BEGIN PRIVATE_VARIABLES */

/* USER CODE END PRIVATE_VARIABLES */

/**
  * @}
  */

/** @defgroup USBD_CUSTOM_HID_Exported_Variables USBD_CUSTOM_HID_Exported_Variables
  * @brief Public variables.
  * @{
  */
extern USBD_HandleTypeDef hUsbDeviceFS;
:

 ※ The HID descriptor could be generated by the program, HID Descriptor Tool, provided by USB org.

      That structure means, for both input(device to host) and output, the packet size are 32 bytes.

     The remain part is to make sure the customHID workable, that is very straightforword : to create the input and output interfaces for relaying the input to the UART, and responding to the output while the button pressed :

USB_DEVICE\App\usbd_custom_hid_if.c :

:
/**
  * @brief  Manage the CUSTOM HID class events
  * @param  event_idx: Event index
  * @param  state: Event state
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */

static int8_t CUSTOM_HID_OutEvent_FS(uint8_t event_idx, uint8_t state)
{
  /* USER CODE BEGIN 6 */
	UNUSED(event_idx);
	UNUSED(state);

	USBD_CUSTOM_HID_HandleTypeDef *hhid
	  = (USBD_CUSTOM_HID_HandleTypeDef *)hUsbDeviceFS.pClassData;

	uint32_t usb_recv_len = USBD_GetRxCount(&hUsbDeviceFS, CUSTOM_HID_EPOUT_ADDR);
	PRINTF("Custom HID::usb_recv_len = %u\r\n", usb_recv_len);
	PRINTF("Custom HID::received data = ");
	for(uint32_t i = 0; i < usb_recv_len; i++)
		PRINTF_NO_COMNAME("%02X ", hhid->Report_buf[i]);
	PRINTF_NO_COMNAME("\r\n");

	return (USBD_OK);
  /* USER CODE END 6 */
}

/* USER CODE BEGIN 7 */
/**
  * @brief  Send the report to the Host
  * @param  report: The report to be sent
  * @param  len: The report length
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
static int8_t USBD_CUSTOM_HID_SendReport_FS(uint8_t *report, uint16_t len)
{
  return USBD_CUSTOM_HID_SendReport(&hUsbDeviceFS, report, len);
}

/* USER CODE END 7 */

/* USER CODE BEGIN PRIVATE_FUNCTIONS_IMPLEMENTATION */

int8_t USBD_CUSTOM_HID_Send(uint8_t data[USBD_CUSTOMHID_OUTREPORT_BUF_SIZE])
{
	return USBD_CUSTOM_HID_SendReport_FS(&data[0], USBD_CUSTOMHID_OUTREPORT_BUF_SIZE);
}

/* USER CODE END PRIVATE_FUNCTIONS_IMPLEMENTATION */

main.c :

:
  uint32_t k = 0;

  while (1)
  {
    /* USER CODE END WHILE */

    /* USER CODE BEGIN 3 */
	  if (true == s_is_GPIO_EXTI)
	  {
		  switch (s_GPIO_Pin)
		  {
		  	  case KEY0_Pin: {
				if (GPIO_PIN_RESET == HAL_GPIO_ReadPin(KEY0_GPIO_Port, KEY0_Pin))
				{
					HAL_GPIO_WritePin(LED0_GPIO_Port, LED0_Pin, GPIO_PIN_RESET);
					PRINTF("KEY0 pushed\r\n");
					uint8_t customhid_data[USBD_CUSTOMHID_OUTREPORT_BUF_SIZE] = {0};
					memcpy(&customhid_data[0], &k, sizeof(uint32_t));
					USBD_CUSTOM_HID_Send(&customhid_data[0]);
					PRINTF("k = %u\r\n", k);
					k++;
				}
				else
				{
					HAL_GPIO_WritePin(LED0_GPIO_Port, LED0_Pin, GPIO_PIN_SET);
					PRINTF("KEY0 released\r\n");
				}
:

To make sure it works, it could use ST's USB HID Demonstrator(installation is required) for testing. 


    The figures show their results are consisted.
    ※ Do not be confused the zeroth byte results, that fields are reserved for the length.

    乙. Bring CDC working

     The parameters on the ico for CDC are all default, we directly use the auto-generated code. But we rename the variable from  USBD_Interface_fops_FS  to USBD_CDC_Interface_fops_FS, as below:

:
// the original name was USBD_Interface_fops_FS  
USBD_CDC_ItfTypeDef USBD_CDC_Interface_fops_FS =
{
  CDC_Init_FS,
  CDC_DeInit_FS,
  CDC_Control_FS,
  CDC_Receive_FS
};
:

that makes the variable meaning be much more clear and explicit while we do the compositing work.

    丙. Bring MSC working

     Change the parameter MSC_MEDIA_PACKET on the ioc, as the integer multiple of your flash sector size. If you are using a common SD card, most of them are sector size 512 bytes (but a little bit is 1024). Here MSC_MEDIA_PACKET is set as 512.

     After the code generated, we need to implement the spi-diskio layer for usbd_storage_if.c using. The implementation is as this post. But we rename the spi-diskio file names from user_diskio_spi to spi-diskio, and the function names from USER_SPI_xx (like, USER_SPI_read) to spi_diskio_xx (like, spi_diskio_read).

      That post is good, that does work. However, one insufficiency needs to be improved : there is the possibility to reset the SD card fail. So it needs to add the retry loop in the diskio initialization: 

inline DSTATUS spi_diskio_initialize (
	BYTE drv		/* Physical drive number (0) */
)
{
	:
	FCLK_SLOW();
	for (n = 10; n; n--) xchg_spi(0xFF);	/* Send 80 dummy clocks */

	ty = 0;

	uint32_t retry_times = 0;
	do
	{
		if (send_cmd(CMD0, 0) == 1) {			/* Put the card SPI/Idle state */
			SPI_Timer_On(1000);					/* Initialization timeout = 1 sec */
			if (send_cmd(CMD8, 0x1AA) == 1) {	/* SDv2? */
				for (n = 0; n < 4; n++) ocr[n] = xchg_spi(0xFF);	/* Get 32 bit return value of R7 resp */
				if (ocr[2] == 0x01 && ocr[3] == 0xAA) {				/* Is the card supports vcc of 2.7-3.6V? */
					while (SPI_Timer_Status() && send_cmd(ACMD41, 1UL << 30)) ;	/* Wait for end of initialization with ACMD41(HCS) */
					if (SPI_Timer_Status() && send_cmd(CMD58, 0) == 0) {		/* Check CCS bit in the OCR */
						for (n = 0; n < 4; n++) ocr[n] = xchg_spi(0xFF);
						ty = (ocr[0] & 0x40) ? CT_SD2 | CT_BLOCK : CT_SD2;	/* Card id SDv2 */
					}
				}
			} else {	/* Not SDv2 card */
				if (send_cmd(ACMD41, 0) <= 1) 	{	/* SDv1 or MMC? */
					ty = CT_SD1; cmd = ACMD41;	/* SDv1 (ACMD41(0)) */
				} else {
					ty = CT_MMC; cmd = CMD1;	/* MMCv3 (CMD1(0)) */
				}
				while (SPI_Timer_Status() && send_cmd(cmd, 0)) ;		/* Wait for end of initialization */
				if (!SPI_Timer_Status() || send_cmd(CMD16, 512) != 0)	/* Set block length: 512 */
					ty = 0;
			}

			break;
		}

		retry_times++;
		SPI_Timer_On(1000);
#define MAX_PUT_SPI_IDLE_STATE_RETRY_TIMES			(100)
	}while(retry_times < MAX_PUT_SPI_IDLE_STATE_RETRY_TIMES);

	//printf("retry_times = %lu\r\n", retry_times);
	CardType = ty;	/* Card type */
	despiselect();
	:

To here, the implementation of the functions at the file USB_DEVICE\App\usbd_storage_if.c is simple, just pass the arguments to spi-diskio, as below :

/* Private functions ---------------------------------------------------------*/
/**
  * @brief  Initializes over USB FS IP
  * @param  lun:
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_Init_FS(uint8_t lun)
{
  /* USER CODE BEGIN 2 */
	spi_diskio_initialize(0);
	return (USBD_OK);
  /* USER CODE END 2 */
}

/**
  * @brief  .
  * @param  lun: .
  * @param  block_num: .
  * @param  block_size: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_GetCapacity_FS(uint8_t lun, uint32_t *block_num, uint16_t *block_size)
{
  /* USER CODE BEGIN 3 */
	spi_diskio_ioctl(0, GET_SECTOR_COUNT, block_num);
	spi_diskio_ioctl(0, GET_SECTOR_SIZE, block_size);
	return (USBD_OK);
  /* USER CODE END 3 */
}

/**
  * @brief  .
  * @param  lun: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_IsReady_FS(uint8_t lun)
{
  /* USER CODE BEGIN 4 */
	return (USBD_OK);
  /* USER CODE END 4 */
}

/**
  * @brief  .
  * @param  lun: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_IsWriteProtected_FS(uint8_t lun)
{
  /* USER CODE BEGIN 5 */
  return (USBD_OK);
  /* USER CODE END 5 */
}

/**
  * @brief  .
  * @param  lun: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_Read_FS(uint8_t lun, uint8_t *buf, uint32_t blk_addr, uint16_t blk_len)
{
  /* USER CODE BEGIN 6 */
	PRINTF("%s :: blk_addr = %lu, blk_len = %u\r\n", __FUNCTION__, blk_addr, blk_len);
	spi_diskio_read(0, buf, blk_addr, blk_len);
	return (USBD_OK);
  /* USER CODE END 6 */
}

/**
  * @brief  .
  * @param  lun: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  */
int8_t STORAGE_Write_FS(uint8_t lun, uint8_t *buf, uint32_t blk_addr, uint16_t blk_len)
{
  /* USER CODE BEGIN 7 */
	PRINTF("%s :: blk_addr = %lu, blk_len = %u\r\n", __FUNCTION__, blk_addr, blk_len);
	spi_diskio_write(0, buf, blk_addr, blk_len);
	return (USBD_OK);
  /* USER CODE END 7 */
}

/**
  * @brief  .
  * @param  None
  * @retval .
  */
int8_t STORAGE_GetMaxLun_FS(void)
{
  /* USER CODE BEGIN 8 */
  return (STORAGE_LUN_NBR - 1);
  /* USER CODE END 8 */
}

/* USER CODE BEGIN PRIVATE_FUNCTIONS_IMPLEMENTATION */

/* USER CODE END PRIVATE_FUNCTIONS_IMPLEMENTATION */
:
 

  Now, the USB mass storage should work, the computer is able to access the SD card via STM32.
The debug uart shows like below (Specifically how the uart shows, depends on what contains in the SD card)


 
貳. Make the composite device.
   
      甲. The Management of the Handles to the Corresponding USB Device Class.
      The STM32 USB stack uses the different handle to deal with the different USB device class. The operation functions structure types are named as USBD_XXXX_ItfTypeDef, and the working_data  structure types are USBD_XXX_HandleTypeDef; those would be as the members of structure  USBD_HandleTypeDef,  pClassData(for the corresponding operations) and pUserData (the handle, it is the working data) for the host calling response. 
     While there is a composite device, we need the different handles corresponding to the different USB classes of course. Thus it is necessary to implement the handle switching. Also for the handles, those should be static, to keep the data intact while switched to the other handles.
     
    We create the file, USB_DEVICE/App/usbd_composite_handles.h :
#ifndef _USBD_COMPOSITE_HANDLES_H_
#define _USBD_COMPOSITE_HANDLES_H_

#include "usbd_customhid.h"
#include "usbd_cdc.h"
#include "usbd_custom_hid_if.h"
#include "usbd_cdc_if.h"
#include "usbd_msc.h"
#include "usbd_storage_if.h"

USBD_CUSTOM_HID_HandleTypeDef* GetCustomHIDHandlePtr(void);
USBD_CDC_HandleTypeDef* GetCDCHandlePtr(void);
USBD_MSC_BOT_HandleTypeDef* GetMSCBOTHandlePtr(void);

void SwitchHandleInterfaceToCustomHID(USBD_HandleTypeDef *pdev);
void SwitchHandleInterfaceToCDC(USBD_HandleTypeDef *pdev);
void SwitchHandleInterfaceToMSC(USBD_HandleTypeDef *pdev);

#endif /* _USBD_COMPOSITE_HANDLES_H_ */

 The implementation of the device swithing is simple. But beware, it is necessary to switch operations and the handle at the same time, as USB_DEVICE/App/usbd_composite_handles.c :
#include "usbd_composite_handles.h"

static USBD_CUSTOM_HID_HandleTypeDef s_p_custom_hid_handle;
static USBD_CDC_HandleTypeDef s_cdc_handle;
static USBD_MSC_BOT_HandleTypeDef s_msc_bot_handle;

/*******************************************************************************/

USBD_CUSTOM_HID_HandleTypeDef* GetCustomHIDHandlePtr(void){ return &s_p_custom_hid_handle;}
USBD_CDC_HandleTypeDef* GetCDCHandlePtr(void){ return &s_cdc_handle;}
USBD_MSC_BOT_HandleTypeDef* GetMSCBOTHandlePtr(void){ return &s_msc_bot_handle;}
/*******************************************************************************/

void SwitchHandleInterfaceToCustomHID(USBD_HandleTypeDef *pdev)
{
	pdev->pUserData = &USBD_CustomHID_fops_FS;
	pdev->pClassData = &s_p_custom_hid_handle;
}

:

 
In the STM USB stack, the handles are created by malloc; in baremetal, there is the static_malloc function in the file usbd_conf.c,  which returns the static handles. We modify the static_malloc function to be able return the 3 different-type instances : 
:
void *USBD_static_malloc(uint32_t size)
{
	if(size == sizeof(USBD_CUSTOM_HID_HandleTypeDef))
		return GetCustomHIDHandlePtr();
	if(size == sizeof(USBD_CDC_HandleTypeDef))
		return GetCDCHandlePtr();
	if(size == sizeof(USBD_MSC_BOT_HandleTypeDef))
		return GetMSCBOTHandlePtr();

	return NULL;
}
:

      乙. The USB Device Operation Interface (USBD_ClassTypeDef) and Composite Device Configure Descriptor.
 
Create the file, USB_DEVICE\App\usbd_customhid_cdc_msc_composite.h, as 
#ifndef _USBD_CustomHID_CDC_MSC_COMPOSITE_H_
#define _USBD_CustomHID_CDC_MSC_COMPOSITE_H_

#include "usbd_customhid.h"
#include "usbd_custom_hid_if.h"
#include "usbd_cdc.h"
#include "usbd_cdc_if.h"
#include "usbd_msc.h"
#include "usbd_storage_if.h"

extern USBD_ClassTypeDef  USBD_CustomHID_CDC_MSC_COMPOSITE;

#endif
It is obvious this handle is to replace the single class handle for USB initialization.

For USB_DEVICE\App\usbd_customhid_cdc_msc_composite.c, there is the configure descriptor. take care,  the order is CUSTOM_HID, CDC then MSC. The CUSTOM_HID must be ahead or it would not work (but I do not know why).

#define USBD_IAD_DESC_SIZE					0x08
#define USBD_IAD_DESCRIPTOR_TYPE			0x0B


#ifdef USBD_MAX_NUM_INTERFACES
	#undef USBD_MAX_NUM_INTERFACES
#endif
#define USBD_MAX_NUM_INTERFACES				4 /*1 for Custom HID, 2 for CDC, 1 for MSC */

#define USBD_CUSTOM_HID_INTERFACE_NUM		1	/* CUSTOM_HID Interface NUM */
#define USBD_CUSTOM_HID_INTERFACE 			0

#define USBD_CDC_INTERFACE_NUM				2	/* CDC Interface NUM */
#define USBD_CDC_FIRST_INTERFACE			1	/* CDC FirstInterface */
#define USBD_CDC_CMD_INTERFACE				1
#define USBD_CDC_DATA_INTERFACE				2

#define USBD_MSC_INTERFACE_NUM				1	/* MSC Interface NUM */
#define USBD_MSC_FIRST_INTERFACE			3
#define USBD_MSC_INTERFACE					3


#define CUSTOM_HID_INDATA_NUM				(CUSTOM_HID_EPIN_ADDR & 0x0F)
#define CUSTOM_HID_OUTDATA_NUM				(CUSTOM_HID_EPOUT_ADDR & 0x0F)

#define CDC_INDATA_NUM						(CDC_IN_EP & 0x0F)
#define CDC_OUTDATA_NUM						(CDC_OUT_EP & 0x0F)
#define CDC_OUTCMD_NUM						(CDC_CMD_EP & 0x0F)

#define MSC_INDATA_NUM						(MSC_EPIN_ADDR & 0x0F)
#define MSC_OUTDATA_NUM						(MSC_EPOUT_ADDR & 0x0F)

__ALIGN_BEGIN uint8_t USBD_CustomHID_CDC_MSC_Composite_CfgFSDesc[USBD_CUSTOMHID_CDC_MSC_COMPOSITE_DESC_SIZE] __ALIGN_END =
{
	0x09,   /* bLength: Configuation Descriptor size */
	USB_DESC_TYPE_CONFIGURATION,   /* bDescriptorType: Configuration */
	WBVAL(USBD_CUSTOMHID_CDC_MSC_COMPOSITE_DESC_SIZE),
	USBD_MAX_NUM_INTERFACES ,  /* bNumInterfaces: */
	0x01,   /* bConfigurationValue: */
	0x00,   /* iConfiguration: */
	0xC0,   /* bmAttributes: self powered */
	0x00,   /* MaxPower 0 mA */


	/***************************CUSTOM HID********************************/
	/*Interface Descriptor */
	0x09,         /*bLength: Interface Descriptor size*/
	USB_DESC_TYPE_INTERFACE,/*bDescriptorType: Interface descriptor type*/
	USBD_CUSTOM_HID_INTERFACE,         /*bInterfaceNumber: Number of Interface*/
	0x00,         /*bAlternateSetting: Alternate setting*/
	0x02,         /*bNumEndpoints*/
	0x03,         /*bInterfaceClass: CUSTOM_HID*/
	0x00,         /*bInterfaceSubClass : 1=BOOT, 0=no boot*/
	0x00,         /*nInterfaceProtocol : 0=none, 1=keyboard, 2=mouse*/
	0x00,  /*iInterface: Index of string descriptor*/

	/******************** Descriptor of CUSTOM_HID *************************/
	0x09,         /*bLength: CUSTOM_HID Descriptor size*/
	CUSTOM_HID_DESCRIPTOR_TYPE, /*bDescriptorType: CUSTOM_HID*/
	0x11,         /*bCUSTOM_HIDUSTOM_HID: CUSTOM_HID Class Spec release number*/
	0x01,
	0x00,         /*bCountryCode: Hardware target country*/
	0x01,         /*bNumDescriptors: Number of CUSTOM_HID class descriptors to follow*/
	0x22,         /*bDescriptorType*/
	USBD_CUSTOM_HID_REPORT_DESC_SIZE,/*wItemLength: Total length of Report descriptor*/
	0x00,
	:


	/****************************CDC************************************/
	/* Interface Association Descriptor */
	USBD_IAD_DESC_SIZE,               // bLength
	USBD_IAD_DESCRIPTOR_TYPE,         // bDescriptorType
	USBD_CDC_FIRST_INTERFACE,         // bFirstInterface
	USBD_CDC_INTERFACE_NUM,           // bInterfaceCount
	0x02,                             // bFunctionClass
	0x02,                             // bFunctionSubClass
	0x01,                             // bInterfaceProtocol
	0x00,                             // iFunction

	/*Interface Descriptor */
	0x09,   /* bLength: Interface Descriptor size */
	USB_DESC_TYPE_INTERFACE,  /* bDescriptorType: Interface */
	/* Interface descriptor type */
	USBD_CDC_CMD_INTERFACE,   /* bInterfaceNumber: Number of Interface */
	0x00,   /* bAlternateSetting: Alternate setting */
	0x01,   /* bNumEndpoints: One endpoints used */
	0x02,   /* bInterfaceClass: Communication Interface Class */
	0x02,   /* bInterfaceSubClass: Abstract Control Model */
	0x01,   /* bInterfaceProtocol: Common AT commands */
	0x00,   /* iInterface: */
	:

	/********************  Mass Storage interface ********************/
	0x09,   /* bLength: Interface Descriptor size */
	USB_DESC_TYPE_INTERFACE,   /* bDescriptorType: */
	USBD_MSC_INTERFACE,   /* bInterfaceNumber: Number of Interface */
	0x00,   /* bAlternateSetting: Alternate setting */
	0x02,   /* bNumEndpoints*/
	0x08,   /* bInterfaceClass: MSC Class */
	0x06,   /* bInterfaceSubClass : SCSI transparent*/
	0x50,   /* nInterfaceProtocol */
	0x05,          /* iInterface: */

	/********************  Mass Storage Endpoints ********************/
	:
};


In the same file, there are the operations for the composite device. The implementation is : to use the endpoint address to determine what is class under process,  then switch to the corrosponding handle and operations, and call the the original class process routine, as below.

static uint8_t USBD_CustomHID_CDC_MSC_Composite_Init(USBD_HandleTypeDef *pdev,
		uint8_t cfgidx);

static uint8_t USBD_CustomHID_CDC_MSC_Composite_DeInit(USBD_HandleTypeDef *pdev,
		uint8_t cfgidx);

:
static uint8_t USBD_CustomHID_CDC_MSC_Composite_Setup (USBD_HandleTypeDef *pdev, USBD_SetupReqTypedef *req);
:
USBD_ClassTypeDef USBD_CustomHID_CDC_MSC_COMPOSITE = { USBD_CustomHID_CDC_MSC_Composite_Init, USBD_CustomHID_CDC_MSC_Composite_DeInit, USBD_CustomHID_CDC_MSC_Composite_Setup,
:
USBD_CustomHID_CDC_MSC_Composite_GetDeviceQualifierDescriptor, }; : /*******************************************************************************/ static uint8_t USBD_CustomHID_CDC_MSC_Composite_Init (USBD_HandleTypeDef *pdev, uint8_t cfgidx) { uint8_t res = 0; pdev->pUserData = &USBD_CustomHID_fops_FS; res += USBD_CUSTOM_HID.Init(pdev, cfgidx); pdev->pUserData = &USBD_CDC_Interface_fops_FS; res += USBD_CDC.Init(pdev, cfgidx); pdev->pUserData = &USBD_Storage_Interface_fops_FS; res += USBD_MSC.Init(pdev,cfgidx); return res; } : /*******************************************************************************/ static uint8_t USBD_CustomHID_CDC_MSC_Composite_DataIn(USBD_HandleTypeDef *pdev, uint8_t epnum) { switch(epnum) { case CUSTOM_HID_INDATA_NUM: SwitchHandleInterfaceToCustomHID(pdev); return(USBD_CUSTOM_HID.DataIn(pdev, epnum)); case CDC_INDATA_NUM: SwitchHandleInterfaceToCDC(pdev); return(USBD_CDC.DataIn(pdev, epnum)); case MSC_INDATA_NUM: SwitchHandleInterfaceToMSC(pdev); return(USBD_MSC.DataIn(pdev, epnum)); default: break; } return USBD_FAIL; }


Below are the address allocation of the 3 classes endpoints.
:
/** @defgroup USBD_CUSTOM_HID_Exported_Defines
* @{ */ #define CUSTOM_HID_EPIN_ADDR 0x81U #define CUSTOM_HID_EPIN_SIZE 0x02U #define CUSTOM_HID_EPOUT_ADDR 0x01U #define CUSTOM_HID_EPOUT_SIZE 0x02U #define USB_CUSTOM_HID_CONFIG_DESC_SIZ 41U #define USB_CUSTOM_HID_DESC_SIZ 9U /*---------- -----------*/ #define USBD_CUSTOMHID_OUTREPORT_BUF_SIZE 32
:

usbd_cdc.h (under Middlewares\ST\STM32_USB_Device_Library\Class\CDC\Inc\): 
:
/** @defgroup usbd_cdc_Exported_Defines
  * @{
  */
#define CDC_IN_EP                                   0x82U  /* EP for data IN */
#define CDC_OUT_EP                                  0x02U  /* EP for data OUT */
#define CDC_CMD_EP                                  0x84U  /* EP for CDC commands */
:

:
#define MSC_EPIN_ADDR                0x83U
#define MSC_EPOUT_ADDR               0x03U
:


      丙. Allocate the PMA
      In STM32, there is a buffer, called Packet Memory Area(PMA) for store the USB packet. 
   the allocation of PMA should begins from the endpoints addresses, next, the packet buffer for each endpoint, in the order of the endpoint address. To avoid the data disordered, the buffer for the endpoints could not overlap each other definitely. 
      What is the total PMA size? it depends on what MCU we use. In the STM32F103 user manual, page 34, it reads :
      shared 512 byte USB/CAN SRAM.
     Thus, the using amount of the PMA buffer could not be more than 512 B.      
     we allocate the PMA buffer as the below figure.

The correpending code to the allocation is as below ( the function USBD_LL_Init in  USB_DEVICE\Target\usbd_conf.c):
:
USBD_StatusTypeDef USBD_LL_Init(USBD_HandleTypeDef *pdev)
{
  :
  HAL_PCD_RegisterIsoInIncpltCallback(&hpcd_USB_FS, PCD_ISOINIncompleteCallback);
#endif /* USE_HAL_PCD_REGISTER_CALLBACKS */

  /* USER CODE BEGIN EndPoint_Configuration */

  /*
   * Buffer Description Table
   * https://www.docin.com/p-760532785.html
   * https://blog.csdn.net/wangzibigan/article/details/54409734
   * https://www.amobbs.com/thread-5692754-1-1.html
   */
#define ACTUAL_USED_ENDPOINTS				(1 + 1 + 1 + 1 + 1)
  uint32_t pma_adress = BTABLE_ADDRESS + (ACTUAL_USED_ENDPOINTS) * (4 + 4);

  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, 0x00 , PCD_SNG_BUF, pma_adress); /*EP0_IN = RX*/
  pma_adress += USB_FS_MAX_PACKET_SIZE;
  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, 0x80 , PCD_SNG_BUF, pma_adress); /*EP0_OUT = TX*/
  pma_adress += USB_FS_MAX_PACKET_SIZE;

  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, CUSTOM_HID_EPIN_ADDR , PCD_SNG_BUF, pma_adress);
  pma_adress += USBD_CUSTOMHID_OUTREPORT_BUF_SIZE;
  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, CUSTOM_HID_EPOUT_ADDR, PCD_SNG_BUF, pma_adress);
  pma_adress += USBD_CUSTOMHID_OUTREPORT_BUF_SIZE;

  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, CDC_IN_EP, PCD_SNG_BUF, pma_adress);
  pma_adress += CDC_DATA_FS_MAX_PACKET_SIZE;
  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, CDC_OUT_EP, PCD_SNG_BUF, pma_adress);
  pma_adress += CDC_DATA_FS_MAX_PACKET_SIZE;

  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, MSC_EPIN_ADDR , PCD_SNG_BUF, pma_adress);
  pma_adress += MSC_MAX_FS_PACKET;
  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, MSC_EPOUT_ADDR , PCD_SNG_BUF, pma_adress);
  pma_adress += MSC_MAX_FS_PACKET;

  HAL_PCDEx_PMAConfig((PCD_HandleTypeDef*)pdev->pData, CDC_CMD_EP, PCD_SNG_BUF, pma_adress);
  pma_adress += CDC_CMD_PACKET_SIZE;

  /* USER CODE END EndPoint_Configuration */
  return USBD_OK;
}
:

     If you are interested in how to set the allocation of PMA for non-STM32F1, please refer to this code, which demonstrates how to set PMA by calling the functions, HAL_PCDEx_SetRxFiFo/HAL_PCDEx_SetTxFiFo.
     丁. The Composite Device Initialization and the device descriptors.
      Similiar with the single class device, we register the operation interface USBD_CustomHID_CDC_MSC_COMPOSITE into the USB device handle, and start that handle.

void USB_DEVICE_CustomHID_CDC_MSC_Composite_Init(void)
{
	if(0x00 != hUsbDeviceFS.pDesc)
		USBD_DeInit(&hUsbDeviceFS);
	USB_DEVICE_Renumerate();

	if(USBD_Init(&hUsbDeviceFS, &FS_Desc, DEVICE_FS)!= USBD_OK)
		Error_Handler();
	if(USBD_RegisterClass(&hUsbDeviceFS, &USBD_CustomHID_CDC_MSC_COMPOSITE) != USBD_OK)
		Error_Handler();
	if (USBD_Start(&hUsbDeviceFS) != USBD_OK)
	    Error_Handler();
}

※ The function USB_DEVICE_Renumerate is to force the host re-enumerate the USB device. It means, while the USB device class changes, like, from CDC -> CustomHID + CDC+ MSC, it is not necessary to restart the firmware. More detail please read here.

  To correctly get the device destriptor and strings, it is necesary to slightly modify the functions set in the operation structure USBD_DescriptorsTypeDef FS_Desc, But the modification is simple, we leave it out.

  One more issue needs to be heeded : here we adopt the pid and strings of the MSC instead of the CDC. as below code :

/** @defgroup USBD_DESC_Private_Defines USBD_DESC_Private_Defines
  * @brief Private defines.
  * @{
  */

#define USBD_VID     1155
#define USBD_LANGID_STRING     1033
#define USBD_MANUFACTURER_STRING     "STMicroelectronics"

/*
#define USBD_PID_FS     22352
#define USBD_PRODUCT_STRING_FS     "STM32 Custom Human interface"
#define USBD_CONFIGURATION_STRING_FS     "Custom HID Config"
#define USBD_INTERFACE_STRING_FS     "Custom HID Interface"
*/

/*
#define USBD_PID_FS     22336
#define USBD_PRODUCT_STRING_FS     "STM32 Virtual ComPort"
#define USBD_CONFIGURATION_STRING_FS     "CDC Config"
#define USBD_INTERFACE_STRING_FS     "CDC Interface"
*/


#define USBD_PID_FS     22314
#define USBD_PRODUCT_STRING_FS     "STM32 Mass Storage"
#define USBD_CONFIGURATION_STRING_FS     "MSC Config"
#define USBD_INTERFACE_STRING_FS     "MSC Interface"


/* USER CODE BEGIN PRIVATE_DEFINES */
      Why do we use the MSC pid and strings ? It is, if we use the (STM32CubeIDE auto-generated) CDC's pid and strings, while the STM32 CDC driver installed, the USB composite device will not work : it is because the STM32 CDC driver would assume the USB device has only one USB class, not a composite device. So, while the pid is MSC, the installed STM32 driver would not activate, and the default Windows CDC driver would be adopted to deal with our composite device and work fine.

   丙. Run the USB Three Classese Composite Device.

    After code build, download the binary into STM32, and there will be a composite device, with CDC, custom HID amd MSC,  shows on the device manager :