My AI Illustration Workflow

Over time I’ve been slowly improving the workflow I use to generate AI illustrations for these newsletters.

In the past I would craft my own prompts for the Bing image generator or the AI image generator built into Substack. But I found that I wasn’t especially good at coming up with good prompts reliably.

I’ve found that Claude is much better than I am at prompt engineering. So for a while I would copy my articles into Claude and ask it to suggest image prompts for the article at several levels of abstractness. I’d then copy the prompts back into Substack’s image generator and see which results I liked best. This worked better, but was quite tedious to do for each article.

Currently I use a local shell script to both generate the prompts and then generate illustrations from these prompts for me to choose from.

1. I run the script in my terminal.
2. The script asks me for the text of my article. I paste the whole article, and add “END” on the next line to signal the script to proceed.
3. The script sends a request to Claude Haiku 3 to extract key themes and concepts from the article.
4. The script loops through a list of styles, asking Claude Haiku 3 to produce an image prompt for each style given the themes and concepts extracted in step three. It makes two such requests for each style.
5. The script pauses to allow me to review and approve the generated prompts.
6. Once approved, the script proceeds to generate an image for each prompt using Z-Image-Turbo running on my local machine.
7. I review the resulting images and select one to be used as the article’s illustration.

I’ve found this workflow to reliably and efficiently produce high-quality illustrations for my articles at very low cost. The Haiku 3 requests cost me less than a penny per run, and the actual image generation is ~free since it runs on my local machine.

One limitation of this approach is that it requires you have a computer beefy enough to run an image generator locally. I have an NVIDIA GeForce RTX 3060 with 12GB VRAM which makes this practical. But without a dedicated graphics card I’d probably be back to using a hosted image generator, perhaps using OpenAI’s API. But of course that would mean higher cost per run.

Here are the illustrations that were generated for the above text:

And these are the prompts that produced them, from bottom right to top left:

1. “Oil painting of a human hand and an artificial intelligence robotic hand clasped together against a backdrop of interconnected gears, circuit boards, and computer components. The palette features muted, earthy tones with highlights of cool metallic grays and blues, conveying a sense of collaboration and technological integration. Visible, impasto brushstrokes create a textured, documentary-style surface, and the composition is balanced with a focus on the central intertwined hands. Dramatic, raking light casts strong shadows, emphasizing the depth and materiality of the scene.”
2. “Oil painting of a human hand and an AI-generated robotic hand reaching towards each other, their fingertips almost touching. The hands are rendered in a highly realistic, detailed style with visible brush strokes and a muted, earthy color palette of browns, ochres, and grays. The background is blurred and hazy, creating a sense of depth and focus on the central interaction between the human and machine. Subtle highlights and shadows emphasize the texture of the skin and metal, while the composition creates a sense of tension and collaboration between the two entities.”
3. “Oil painting of a human hand collaborating with a metallic, robotic hand emerging from a swirling cloud of colorful digital data. The composition is balanced and symmetrical, with the human and robotic hands meeting at the center, fingers intertwined. The palette features rich, earthy tones of brown, ochre, and sienna, contrasted by vibrant splashes of blues, greens, and purples that suggest the flow of digital information. The brushwork is smooth and blended, with visible yet controlled impasto texture that adds depth and tactility to the painting. Warm, directional lighting from the upper left casts dramatic shadows and highlights the collaborative nature of the hands, emphasizing the harmony and synergy between human and machine.”
4. “Oil painting of an abstract representation of the collaborative creative process between human and AI. The canvas depicts an energetic, swirling composition of overlapping shapes and gestural brushstrokes in a harmonious color palette of warm ochres, deep blues, and muted greens. Visible, expressive brushwork creates a sense of movement and dynamism, evoking the iterative nature of prompt engineering and workflow optimization. The overall mood is contemplative yet resolute, reflecting the pragmatic and solution-oriented approach described in the article. Areas of the canvas feature sharper, more defined forms juxtaposed with areas of loose, ethereal brushwork, symbolizing the interplay between human intent and machine generation. The painting cultivates a sense of collaboration, experimentation, and the gradual refinement of a creative process.”
5. “Oil painting of an abstract, impressionistic representation of the collaborative interplay between human and AI in the creative process. Warm, earthy tones of ochre, sienna, and umber predominate, with flashes of vibrant primary colors symbolizing the sparks of innovation. Loose, expressive brushstrokes convey a sense of fluidity and dynamism, as if the paint itself is a physical manifestation of the iterative workflow. Subtle textures and blended edges create an atmospheric, dreamlike quality, evoking the experimental nature of the process. The composition features geometric shapes and forms that suggest both structure and flexibility, hinting at the technological and conceptual frameworks underlying the AI-assisted illustration generation. An overall sense of balance and harmony emerges, reflecting the synergy between human intuition and machine intelligence.”
6. “Oil painting with visible brushstrokes depicting an abstract, swirling composition of vibrant colors. Shades of blues, greens, and golds flow together, creating a sense of movement and energy. Pockets of light and shadow weave through the canvas, adding depth and dimension. The overall impression is one of experimentation, iteration, and the harmonious collaboration between human and machine.”
7. “Oil painting with visible brushstrokes depicting a surreal, semi-representational landscape. Muted earthy tones of ochre, sienna, and umber, accented with hints of cool blues and greens. Textured, impasto surfaces create a sense of depth and materiality. Fluid, gestural forms suggest the interplay of human and machine elements, with abstract shapes and symbols hinting at the collaborative creative process. Soft, diffused lighting casts a contemplative, experimental mood over the scene.”
8. “Oil painting with visible brushstrokes depicting an abstract representation of the collaborative creative process between human and AI. Muted, earthy color palette of ochres, siennas, and deep blues, evoking a sense of pragmatism and technical exploration. Textured, impasto surfaces suggest the iterative, experimental nature of the workflow, while gestural, semi-representational forms symbolize the interplay of human intent and machine generation. Warm, soft lighting casts a contemplative atmosphere, hinting at the gradual refinement and optimization of the process over time.”
9. “Oil painting with heavy impasto texture depicting an abstract, symbolic representation of the collaborative creative process between human and AI. Muted earth tones of ochre, sienna, and umber with flashes of metallic gold and silver, conveying a sense of technical sophistication and pragmatic problem-solving. Amorphous, gestural shapes and forms intertwine, suggesting the iterative nature of prompt engineering and workflow optimization. Visible, energetic brushstrokes create a sense of movement and experimentation. The composition is balanced but with areas of visual tension, mirroring the challenges and breakthroughs described in the article. Soft, directional lighting casts shadows that add depth and a contemplative atmosphere.”
10. “Oil painting with impasto technique depicting abstract symbolic elements representing the interplay between human and AI in the creative process. Muted earthy tones of ochre, sienna, and umber with touches of metallic gold and silver accents. Textured, gestural brushstrokes convey a sense of motion and collaboration. Dominant central form suggests a collaborative entity, with surrounding fragmented shapes and lines hinting at the iterative workflow. Hints of technical diagrams or schematics in the background allude to the logistical considerations. Atmospheric lighting casts dramatic shadows, evoking a contemplative, experimental mood.”
11. “Oil painting with a surrealist, metaphorical style. In the center, a large, organic shape resembling an open book or unfolding petals, rendered in warm, earthy tones. Within this shape, fragmented geometric forms in shades of blue, purple, and metallic accents float and intersect, suggesting a complex technological or digital landscape. The background is hazy and atmospheric, with loose, textured brushstrokes in muted greens, ochres, and grays, creating a sense of depth and mystery. The overall composition conveys a balance between the natural and the artificial, the human and the machine, in an enigmatic and thought-provoking manner.”
12. “Oil painting of a surreal, dreamlike landscape where organic forms and geometric shapes coexist in a harmonious balance. The composition features a central focal point of a large, abstract shape resembling a human figure, rendered in muted earth tones and textured brushwork. Surrounding this central element are swirling, ethereal forms in shades of blue and green, suggesting the flow of energy and ideas. Subtle hints of warm ochre and amber tones create an atmosphere of contemplation and introspection. The overall style is dreamlike and evocative, with a sense of the subconscious and the interplay between the human mind and the creative potential of technology.”
13. “Oil painting with a vibrant, geometric abstract composition. Distinct angular shapes in a harmonious color palette of deep crimson, burnt sienna, and vivid cobalt blue. Visible, energetic brushstrokes create a sense of movement and dynamism. Subtle gradients and blended edges add depth and visual interest. The overall mood is one of balance, innovation, and the collaborative embrace of human-AI creativity.”
14. “Oil painting with heavy impasto texture and geometric abstract forms in a vibrant, harmonious color palette of deep blues, rich ochres, and vivid greens. The composition features overlapping angular shapes that create a sense of depth and movement, evoking the collaborative and iterative nature of the illustration workflow. Visible, expressive brushstrokes convey a sense of energy and dynamism, reflecting the experimental and problem-solving approach described in the article. The overall mood is one of intentional, deliberate abstraction, capturing the essence of the themes around prompt engineering and human-AI partnership in a visually striking manner.”
15. “Oil painting with a vibrant, layered color field effect. Warm hues of orange, yellow, and red blended seamlessly, creating a sense of radiant energy. Hints of cool blues and greens emerge from the depths, generating a harmonious interplay of complementary tones. The surface exhibits a thick, impasto texture with visible, expressive brushstrokes that add depth and movement to the composition. The overall atmosphere is one of balance, exploration, and the synergistic collaboration between human and machine.”
16. “Oil painting with a vibrant, energetic color palette of deep blues, fiery oranges, and electric greens. Loose, expressive brushstrokes create a sense of movement and dynamism across the canvas. Layers of transparent washes and opaque impasto textures build depth and visual interest. The composition features a central focal point with bursts of intersecting lines and shapes that radiate outward, evoking a sense of exploration and discovery. Subtle hints of underlying geometric forms suggest the technical and analytical nature of the creative workflow, while the overall atmospheric quality captures the experimental and iterative spirit of the process. The painting exudes a balance of human touch and machine-generated precision, mirroring the collaborative partnership between artist and AI.”