FPGA/CPLD VGA Generator and Shared Memory Access

I am thinking about designing a single board computer that would be capable
of generating VGA signals to display images stored in SRAM on the monitor.
The VGA generator would be implemented in CPLD or FPGA, and it would share
the memory with the processor. Basically, somewhere in the CPU's address
space, I would have a location where the screen image would be stored. The
idea is that the screen image would be accessible by both, the CPU and the
FPGA. The FPGA generator would constantly read from memory the pixel values
and display them on the monitor. However, the CPU should also have the
read/write access to the screen image so that it could be modified.

How do I implement all that so that everything works and doesn't collide?
How do I prevent the CPU and FPGA from accessing the memory at the same
time?

Any comments/feedback would be highly appreciated.

Krystian