Storing a pointer in an integer

Firmware stores buffer addresses in a uint32_t field. It works on the MCU but crashes in unit tests on a 64-bit PC. What is the fix?

struct job { uint32_t buf_addr; uint16_t len; };
job.buf_addr = (uint32_t)buf;
uint8_t *p = (uint8_t *)job.buf_addr;