one must imagine sisyphus full of ideas: on exercising the idea muscle
Lore
It was mid-May. I was a couple of weeks into catching up on post-training literature and learning reinforcement learning for LLMs in a more bottom-up way. Having done some basic experiments, I was unable to come up with an idea that had the allure to possess me for a week or two. I waited as if I was entitled to receive the idea from the muses.
I wondered - if I can be creative in other areas like posting or coming up with ideas when given an objective, why am I not getting good ideas in this specific area of interest. I slowly realised that after years of working on other people's objectives, dismissing building my own projects because they sounded cliche (everyone was doing them); letting small sparks of ideas die because I did not have the energy to protect them, I may have let my idea muscle atrophy.
Don't get me wrong. I do get ideas. Sometimes I get very good ideas and I execute on them for days or weeks on end. It's just not as consistent as I would want it to be. When I say good idea, I don't necessarily mean something novel or groundbreaking. I just mean something that is creative and useful, which can count as a good to great contribution if executed upon well. Maybe it leads to something fun, etc. A good next experiment.
I also realised that in research, coming up with the next good set of ideas, and many of them, matters a lot. I need to start cultivating this muscle better. This led me to contemplate and read more on idea generation/creativity. I am also generally interested in the topic of creativity and since I was on a break, it was a thing to do.
Moreover, intelligence is commoditised (at least in knowledge and coding work due to AI). Agency and taste are more important than ever (I think we have consensus on this now?). Good ideas are the next great adventure. After all, without a good idea what use is agency?
So now here we are - writing a post to record my observations and learnings. Making my problem everyone's problem. I also write this in the hope that some of you (and future me) will execute on my good ideas better than me.
The basics and where we go wrong
The folk recipe for idea generation is simple - do consumption-maxxing, marinate on it for days, walk around, get caffeinated, yap with friends over it and wait for the muse to whisper. I wondered if it could be a bit more systematic? Did the muse have an API? Can I at least learn how to bounce back from dull states to generative states? This was my "struggle". Most of this post is my attempt at making it a bit more systematic.
Most of us do the mistake of consuming in excess and call it thinking. We don't spend enough time processing or applying new information. Perhaps I was more study heavy too than code/experiment heavy (narrator voice: indeed he was.). I liked this particular quote -
After you read something, watch something, listen to something — before you look at what anyone else thinks about it — write down what you actually thought. Not what you think you're supposed to think. Not the smart take. Write down what you actually noticed, what bothered you, what you believed before and whether it changed. Source
Maybe I should consider doing an exercise regularly where I get away from screens and music and just try to come up and write down ideas on paper for 5 minutes. Maybe I am not taking out time to think on things, rushing from one thing to another and constantly context switching.
Good inputs and direct exposure to the source
LLMs make it dangerously easy to live one layer above the source. They are not reading code, using AI to write, using AI to read through long content. If you want to get good ideas and you are a serious people (Logan.jpeg), then you need to go through the content you have selected to go through yourself. Then talk about / think about it a bit. On a meta level, the idea is to consume good inputs in high resolution to build a good embedding space of knowledge. If you have a good embedding space, it will eventually light up when you come across a relevant problem statement and lead to a generative state.
Discovery happens in the doing
Common thing we mess up is waiting too long before applying the newly learnt inputs (or ideas). You have to get to applying your ideas as early as possible, even the silly ones. Get into a feedback loop. This reveals gaps in your understanding and feeds into your thinking better. Once you start, the work begins to feel like a quest. One experiment leads to another. You run into new problems, notice new angles, and sometimes find people who care about the same things, especially if you talk about your work online and offline.
You will hear AI researchers reproduce papers or (even engineers) trying to at least get things running. This helps to identify the details that were not mentioned in source material and also finding gaps in your own understanding.
You have to do a lot and then consume a lot. A common mistake is being consumption heavy and too light on doing. Another mistake is that if you get into doing and are making progress, you may get tunnel visioned by doing and delay learning relevant things that you should be following up on. There is a sweet spot and I think you should definitely lean towards doing. Doing and getting feedback from tight loops is more psychoactive in general. Doing/creating leads to more feedback from reality, people and subsequently more mini problem statements. Sometimes, it may help you discover who you are.
Do more of things that make you feel alive. Follow the things that give you energy after you do them for sometime. Some activities drain you while others return you to yourself.
It's a matter of time (we often underestimate the effort)
Idea generation in its pure essence is a matter of putting in the time. In my case, the answer was less mystical. I was simply early into the inputs for learning RL and another thing is I was approaching it in a bottom up way. I was learning and like just need to put more time in it, do more basic experiments (accept that I am still a fool in this and be humble enough and open minded to struggle and learn from basic things), discuss about it with other people, marinate on it, tweak the details and then maybe some creative idea that's worth spending a good deal of time shall emerge. It's similar to how you need to put in some hours in an RPG like Dark Souls 3 or Elden Ring till your build starts to mature and you find the better weapons. Then you start having fun, with a better build and understanding of the game mechanics. Some part of your brain gets freed up from learning for survival and now you have bandwidth for creativity.
Coming back to RL learning, I made 2 mistakes here: I was not humble enough and I was trying to do a speedrun bottom up. If you are attempting a learning speedrun, be more top down oriented or objective oriented.
In general, I find it better to read things when I actually need them, and that usually works for me. The RL learning was a bit of an exception. I started to enjoy the reading, especially going from the PPO and GRPO basics to the latest on-policy distillation literature and then reading a couple of new papers. I also had to refresh a lot of maths, and I figured I could afford to since I was on a break. Normally I lean top-down, especially now with LLMs around. You get a lot done fast that way, but the learning ends up feeling fragmented. I am sure there is a sweet spot somewhere. Like I mentioned earlier, it is easy to get tunnel-visioned by doing and skip the reading you should be catching up on. The sweet spot is staying disciplined enough to actually read and understand the material, and to take pauses.
Ideas come from inside the work
Another thing I learnt which is in the line of "doing things" is that you are more likely to get ideas when you are inside the work. It's like being in the mine and finding precious stones covered in the mud and dirt. The gaps, bottlenecks, missing details, things you encounter while solving blockers, what if I had done X way, was there a better way to do this?
Some attempts to prose aside, this means you are in the loop. The inputs are freshed and processed. You have recently wrestled with the problem. If you are not currently fresh with the inputs and worrying about getting an idea, you are wasting your time. When you are out of the loop, relax. When you will get back, the wheel should get started.
The people are algo driven and the algo is people driven too
I post a lot on X. When it comes to posting, I have been pretty creative often especially when I have had some momentum of posting for few days consistently. However even there, a lot of my posts are inspired from the discourse (which objectively is not bad and something that I don't mind until it gets boring). Such is the nature of posting of most people. It's influenced by current algo, current discourse. Both the people and the algo are to blame for it because people are more likely to like what's familiar and the algo would subsequently push more of it. From a meta perspective, the discourse is a scaffold, or you can say a prompt.
This does not just happen in posting, people will write articles, blogs, even make projects based on discourse. High agency college students are very good at it (it's a good thing to have). People will build entire startups, in fact most of the startups probably start this way, either some interesting in discourse (Karpathy tweets about auto-research and 10 auto-research startups come out of stealth). Cursor gains traction or Claude Code gets some success, 5 other startups with 1-2 unique variants pop up.
I have talked to several researchers and engineers and they would often mention that they got inspiration to write a blog or make a project because they thought how a certain product they use works or what would happen if they tweaked x in this product or how can I vary this to solve this problem. They got interested in it and executed upon it within their capability. The meta here is curiosity, pro-activeness to try the tools and lastly the whole activity acting as a scaffold/prompt.
You need a prompt
Ok the point I am coming to here is you need a prompt/scaffold. Ideas don't just pop out of nowhere. You need something against which you can bounce off your mind's embeddings, your mind's substrate such that something in there activates and you get sparks. You need to have a prompt, a scaffold, an objective, a problem statement. The mind needs something specific to push against before it gives you anything back. Ideas need friction. You are not entitled to ideas; you have to work for them.
Something for your mind to push and pull against. The problem statement could be as simple as "I want to make more money" and then you may have ideas downstream of that. The mind has this tendency to develop phosphorescence once you get serious about something. It should be based on what you really want to do. I am tempted to say the more specific the better it is, however specificity leads to the trap of being tunnel visioned. Higher resolution would be a better choice.
Agency is downstream of wanting. To what extent you care about it. The more something matters to you, the more your mind recruits the world around it: people, tools, problems, ideas, coincidences.
Post that, it's just a matter of being in a state where you are sensitive enough (not being overstimulated, long walks, caffeinated etc. whatever works for you) to hear it. This is more self-experimentation driven. What routine, foods, people, activities help you to become more generative. Having good health and routine. Maybe having a side quest along with main quest. When I am in such a state, ideas come more, I talk to more people and do more, and it compounds. This is the closest thing to a system I have found.
I recommend reading Henrik Karlsson's Good ideas and this video about the systems and routines of writers/directors like David Fincher. They tend to have strict routines of creative work. David has this thing where he would try to keep as much order as possible in his day such that his mind has space to think of chaos when he sits to imagine and write.
a problem statement is all you need
I get a lot of ideas when I am strongly pursuing a problem statement (after spending lots of time reading or just building with an agent or trying to solve a specific blocker). In fact this is one of my specialisations: to go unreasonably deep into a particular trench if I have to solve something.
When you are trying to solve a problem or attain an objective for whatever reason, you end up doing pretty much everything I have talked about so far.
when you don't have a prompt, just do things
Sometimes you won't have that prompt/objective/scaffold and in those cases you will have to just keep doing and consuming. Doing things may lead you to discover "goals" you want more than your preconceived goals / the goals your rational mind formulated. Expose yourself to good and diverse inputs, let yourself indulge in your curiosities at times. Consume and do things in the direction that makes you feel alive.
Last month, I was in a funk. I was not excited about anything I used to be excited about. I was recovering from a mild illness and had been out of touch with things that I do regularly. He had suggested to do the silliest thing and you may find aliveness in it (and it kinda worked). I wrote a blog on my favourite kpop songs and got my "doing" momentum back.
people are like portals and mirrors and good conversations can re-arrange your brain (h/t Maja)
Talking to people helps a lot. Bouncing back on each other's existing and new ideas. Letting them ask questions, asking them questions. Try to think from their perspective. If you have a few friends who are familiar with your context, they can often remind you what it's like to be the creative you and that can be a great fuel. I am lucky to have few such friends.
Good conversations with good people can help you to spot blind spots. You can also spread context about yourself so others can help you and you can help them in times of need. You can also collab with others, work with others and get ideas from there. Recommended reading: why some conversations rearrange your brain
Another trait, it took me a while to notice. I noticed the following facts about people who work with the door open or the door closed [cf. "working with the garage door up"]. I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don't quite know what problems are worth working on; all the hard work you do is sort of tangential in importance. He who works with the door open gets all kinds of interruptions, but he also occasionally gets clues as to what the world is and what might be important.
Now I cannot prove the cause and effect sequence because you might say, "The closed door is symbolic of a closed mind." I don't know. But I can say there is a pretty good correlation between those who work with the doors open and those who ultimately do important things, although people who work with doors closed often work harder. Somehow they seem to work on slightly the wrong thing—not much, but enough that they miss fame. Richard Hamming, You and Your Research
Introspect by doing
I love this entry from Henrik Karlsson's blog Two kinds of introspection where he nudges us to introspect by doing. To let the doing reveal the world in us.
When Cave sits down each day from 9 to 5:30 pm—he’s a punctual man—hammering out lyrics, listening for the “supreme spiritual potential that only comes into its true and full being if we abandon all those cherished ideas about who we think we are or are not”, he pretends to be battling with the muses. This is the game he plays with himself. But there are no muses. The voice he hears at his typewriter when he listens carefully and blocks out the opinions of others is the voice of Nick Cave pretending to be the possessive god of art.
He is introspecting, but he’s not introspecting by meditating on his existence or talking to a therapist (though maybe he should); he’s introspecting by acting on the world. He’s introspecting by doing stuff and then observing how it makes him feel—as if he was running a million small experiments trying to figure out the composition of this object that is Nick Cave. He is ascribing the feelings he is hearing to the muses, and that might be a good way to think about it as an artist, but the feelings are his. When the muses tell him to lean into a song, to expand it, it is his taste and experience talking.
Breaking through a block
Changing gears a bit, I have focused on "initial ideas" so far but I think most of it transfers for when you are facing a creative block or just blocked at something with respect to problem. In such situations, taking a break helps. Talking helps. Listing out your assumptions and questioning them helps. Simplify the approach. Reread the problem statement. Reframe what you think you are solving. Often the block is not in the problem, but in the cached shape your mind has given it.
I think the main reason I didn’t realize my problems were solvable was a lack of imagination. I’d never seen anyone solve problems in an agentic way, and I failed to imagine that things I hadn’t seen done could be done by me. Source
Taking a break and long walks work because it allows to free up your mental context and zoom out. You can later see things from a fresh perspective, with maybe looser constraints. You often need to loosen up constraints and slither out of the invisible binds on you. Try to be playful. Personally, I do better when I don't take myself too seriously.
If you have a lot of freedom and not many external motivators like a deadline, then imposing certain constraints may help.
Besides this, now that we use AI agents a lot, we often don't have a full understanding of the system. Consider increasing your resolution by going below the level of abstraction you are currently at, i.e. descend into the data/code/log trenches yourself and sweep the solution search space with help of agent.
Source: Zen and the Art of Machine Learning
This article is partly inspired by Jack Morris's post.
There will be times when you are just genuinely blocked and can't really do anything. I asked one of my AI researcher friends at a big lab about it and they mentioned that they usually have multiple projects going on so if one gets hard-blocked on one project, pick up another. They suggested me to consider increasing breadth when learning something new (study it at different levels of resolution/abstraction, study the foundations and the application etc.).
Things I enjoyed reading
A few things I found helpful or just enjoyed while thinking about all this.
On AI research and engineering
These are more AI research specific and very good reads.
I highly recommend reading Jack's entire post. Another one I had enjoyed recently was Karina Nguyen's post.
Few quotes that had resonated with me:
The gap between “trying random things” and “systematically narrowing hypotheses” separates okay researchers from effective ones. Debugging intuition is learnable and comes with running hundreds of experiments.
The most durable skill in AI research isn’t any particular technical contribution, but the ability to repeatedly identify what matters now and execute quickly. Choosing the right problems to work on is as important as executing them with care.
Some discoveries are downstream of infrastructure—you can’t have certain ideas until you have the systems that make them possible to test. The tooling comes first, then the insight. That’s why often researchers spend time on engineering problems.
On creativity, agency, and people
None of us are stuck because we have no options. We’re stuck because the alternative has stopped being something we can picture for ourselves. And in that gap, many of us lose the ability to imagine different futures at all. Source
"If you reject your own ideas, then the part of the brain that comes up with ideas is going to stop. You just do it and do it and do it, and you sort it out later."
I loved this entry on Riley Walz where he talks about ideas and people; it was an inspiration for this post. He is into it for the fun. When people see him dance on his ideas, they join the dance too.
Conclusion
The periods when I get out of the loop willingly (to relax) or unwillingly, the mistake I make is thinking that ideas will come to me; that the muse will whisper in my ears as if I am entitled to good ideas. Turns out it does not work that way. The good thing once I am back into my generative state by doing and talking to people, crossing through the chasm of initial frustration, I get generative again. I guess the whispers of the muse are what momentum sounds like from inside.
I wrote this blog as a reminder of that. I still have the research idea muscle to build, but at least now I know how to get back to the state where ideas come. If you have recommendations or suggestions for me, I would love to hear them. If you liked reading this post, please share!
Now get back to doing things!
Acknowledgements
Lots of inspiration and ideas bounced back from past me, friends @tokenbender, @thepushkarp, @filterpapi, @sachdh, writers @phokarlsson, @majamediaco, @visakanv, @aadilpickle’s Riley Waltz article, and @jxmnop’s “Art of AI Research” article.