…and I still don’t get it. I paid for a month of Pro to try it out, and it is consistently and confidently producing subtly broken junk. I had tried doing this before in the past, but gave up because it didn’t work well. I thought that maybe this time it would be far along enough to be useful.
The task was relatively simple, and it involved doing some 3d math. The solutions it generated were almost write every time, but critically broken in subtle ways, and any attempt to fix the problems would either introduce new bugs, or regress with old bugs.
I spent nearly the whole day yesterday going back and forth with it, and felt like I was in a mental fog. It wasn’t until I had a full night’s sleep and reviewed the chat log this morning until I realized how much I was going in circles. I tried prompting a bit more today, but stopped when it kept doing the same crap.
The worst part of this is that, through out all of this, Claude was confidently responding. When I said there was a bug, it would “fix” the bug, and provide a confident explanation of what was wrong… Except it was clearly bullshit because it didn’t work.
I still want to keep an open mind. Is anyone having success with these tools? Is there a special way to prompt it? Would I get better results during certain hours of the day?
For reference, I used Opus 4.6 Extended.
The solutions it generated were almost write every time
Did you vibe code this post? 😂
I use it and it works. It doesn’t give you the right result in one shot, but neither does manual coding. You iterate and prompt again and again. In the end, it saves a ton of time. Engineers are definitely going to lose their jobs because fewer people are needed. I know its tough to accept this and people will go through denial. Part of that is saying the AI code is junk. But, you’ll find it can produce junk and quickly fix it into the right solution faster than an engineer can. It sucks, but this is the new reality. The one thing that is cool once you embrace it is that you realize you can customize your favorite apps or even build anything you want from scratch.
It sucks, but this is the new reality.
Sorry mate, but you drank the AI koolaid from Sam Altman and the other tech oligarchs. The reality is that all of the major AI companies are deep in the red, OpenAI isn’t even making a profit with the 200$ subscription.
The only reason people are able to burn thousands of tokens to vibecode their apps is that they don’t have to pay the price for that, the companies are. This money will run out soon and then we will see the real cost for the bigger models.
If a subscription for Claude Code costs 500$ or even 1000$, will companies still pay for it or let actual humans do the work? We will see. I seriously doubt it, and I don’t want to depend on a subscription-based service to do my work while my skills are atrophying. Thank god my employer doesn’t force me to use AI.
Engineers are definitely going to lose their jobs
This kind of fear-mongering is what I despise most about the whole bubble.
I haven’t drank Koolaid. I’m talking from my experience using it in my professional software engineering job where I lead software projects. I’ve built things that used to take 20 weeks in 1 week with Claude. My employer does not really care about the cost of the tokens. And, when they can have one engineer do 20 weeks of work in 1 week, that to them is actually a cost savings. I already ask myself the question … Should I give this task to another engineer or just vibecode it myself?
OpenAI may not survive because they do have financial issues from overspending, but that barely matters. The company with the strongest coding LLM is Anthropic and it doesn’t sound like they’re having financial difficulty. Either way, now that it is clear what is possible, some company will succeed.They have incentives to do it.
Like I said, it will suck for some people, but its hard to deny the reality at this point.
You still need programmers because you need people proficient in programming to be able to tell how to fix the junk that it generates into working code.
customize your favorite apps
can you elaborate?
Github is full of open source apps. Some times the maintainer won’t add a feature you want. You can just clone the repo and ask Claude to do it and then run your own version of it.
It’s a tool that you need to learn. Try some of claude.md files people share online for your programming area as a starter. You still need to review what it does but just asking for it to create tests as it creates code does a lot to improve output.
Also working on some 3d maths.
I’ve used the free versions a bit, but not really to the extent that I’d call it vibe coding. The chat bots often know where to find libraries or per-existing functions that I don’t know. It’s also okay at algorithms for well defined problems, but it often says be careful not to do something I absolutely need to do or visea versa. It’s very hit and miss on debugging. It’ll point out obvious stuff (typos) reliably, and it can do some iteration stuff usually, but it usually doesn’t pick up on other things. Once in a rare while it will impress me by suggesting I look at a particular thing, and I think it manages this better in new chats, but most complex issues fail for it. I use it as a faster stackoverflow, but you need to be able to work through the code yourself, understand what you’re doing, and test that individual steps are doing what they need to do. The bots can’t really do any sort of planning or breaking down a problem into sub-problems, and they really suck at thinking about 3d stuff.
I recently started using Pro to debug a problem I couldn’t solve. The one thing I need from it is an extra insight, a second opinion (because I’m the only developer), and it allowing me to let it read the whole folder helps, it identified a problem I didn’t consider because it’s a file outside of where I was looking.
That’s been my experience. It’s always subtlely wrong, its solutions are hard to maintain, and if you spend too much time with it, it starts forgetting what you said earlier. Managers don’t understand the distinction, they already can’t code well, and only test it in small problems where it’s not context-limited, so they’re amazed.
I have a full pro model for Kiro at work. It does actually work, but we have custom MCP servers for all the internal tools, context on how to use these tools, style guidelines, etc. and then on top of that we have a lot of AI context files in the code base to help the AI understand the code base and make the correct changes.
I’ve been using it on a side project and it works if you know how to constrain it. It does get things wrong a lot. But the big thing about it is doing spec driven development where you give it a write up and it makes a requirements doc and a design doc with a lot of correctness properties in them to follow when generating and making the tasks.
I don’t believe people can vibe code unless they can actually code. It’s a whole different way of coding. I still manually edit what it does a lot.
A lot of people explain it like it’s a brand new junior developer. You need to give it as much context as possible, tell it to exactly what you want, tell it what you don’t want, tell it why, etc. and it still may not listen exactly.
you need to fully be able to program to work with these things, in my experience.
you have to explain what you want very specifically, in precise programming terms.i tried a preview of chatgpt codex and it’s working better than my free version of claude, but codex creates a whole virtual programming environment, you have to connect it to a github repository, then it spins up an instance with tools you include and actually tests the code and fixes bugs before sending it back to you.
but you still need to be able to find the bugs and fix them yourself.oh and i think they work best with python, but i’ve also used ruby and dart and it’s decent.
it’s kinda like a power tool, it’ll definitely help you a lot to fix a car but if you can’t do it with wrenches it won’t help very much.I’ve never been able to program in anything more complex than BASIC and command line batch files, but I’m able to get useful output from Claude.
I’m an IT Infrastructure Manager by trade, and I got there through 20 years of supporting everything from desktop to datacenter including weird use cases like controlling systems in a research lab. On top of that, I’ve gotten under the hood of software in the form of running game servers in my spare time.
What you need to get good programs out of AI boils down to 3 things:
- The ability to teach an entity whose mistakes resemble those of a gifted child where it went wrong a step or ten back from where it’s currently looking.
- The ability to provide useful beta test / debug output regarding programs which aren’t behaving as expected. This does include looking at an error log and having some idea what that error means.
- Comfort using (either executing or compiling depending on the language) source code associated with the language you’re doing things in. This might be as simple as “How do I run a Powershell script or verify that I meet the version and module requirements for the script in question?”, or it might be as complicated as building an executable in Visual Studio. Either way whatever the pipeline is from source to execution, it must be a pipeline you’re comfortable working with. If you’re doing things anywhere outside the IT administration space, it’s reasonable to be looking at Python as the best first path rather than Powershell. Personally, I must go where supported first party modules exist for the types of work I’m developing around. In IT Administration, that’s Powershell.
I’ve made tools which automate and improve my entire department’s approach to user data, device data, application inventory, patch management, vulnerability management, and these are changes I started making with a free product three months ago, and two months back I switched to the paid version.
Programming is sort of like conversation in an alien language. For that reason, if you can give precise instructions sometimes you really can pull something new into existence using LLM coding. It’s the same reason that you could say words which have never been said in that specific order before, and have an LLM translate them to Portuguese.
I always used to talk about how everything in a computer was math, and that what interested me more than quantum computing would be a machine which starts performing the same sorts of operations on words or concepts that computers of that day ('90s and '00s when “quantum” was being slapped on everything to mean “fast” or “powerful”) were doing on math. I said that the best indicator when linguistic computing arrives would be that without ever learning to program, I’d start being able to program. I was looking at “Dragon Naturally Speaking” when I had this idea. It was one of the earliest effective speech to text programs. I stopped learning to program immediately and focused exclusively on learning operations from that point forward.
I’ve been testing the code generation abilities of LLMs for about three years. Within the last six months I feel like I’m starting to see evidence that the associations being made internally by LLMs are complex enough to begin considering them the fulfillment of my childhood dream of a “word computer”.
All the shitty stuff about environment and theft of art is all there too, which sucks, but more because our economic model sucks than because LLMs either do or do not suck. If we had a framework for meeting everybody’s basic needs, this software in its current state has the potential to turn everyone with a passion for grammatical and technical precision into a concept based developer practically overnight.
I have no qualifications to judge the quality of the generated results, yet the generated results are always of great quality.
Do you seriously not realize how out of touch this sounds?
I’ve never been able to program in anything more complex than BASIC and command line batch files, but I’m able to get useful output from Claude.
Chatbots being deemed useful in tasks by people unqualified to make those judgments is a running problem.
In my experience there are three ways to be successful with this tool:
- write something that already exists so it doesn’t need to think
- do all the thinking for it upfront (hello waterfall development)
- work in very small iterations that doesn’t require any leaps of logic. Don’t reprompt when it gets something wrong, instead reshape the code so it can only get it right
The issue with debugging is that it doesn’t actually think. LLMs pattern match to a chain of thought based on signals, not reasoning. For it to debug you need good signals in your code that explicitly tell what it is doing and the LLMs do not write code with that level of observability by default.
Edit: one of my workflows that I had success with is as follows:
- write a gherkin feature file describing desired functionality, maybe have the LLM create multiple scenarios after I defined one to copy from
- tell the LLM to write tests using those feature files, does an okay job but needs help making tests run in parallel.
- if the feature is simple, ask the LLM to make a plan and review it
- if the feature is complex then stub out the implementation in code and add TODOs, then direct the LLM to plan. Giving explicit goals in the code itself reduces token consumption and yield better plans
write something that already exists so it doesn’t need to think
If something already exists, it shouldn’t need to be rewritten.
Doing otherwise is a sign that something has gone wrong.
That was the case before LLMs and it is still the case today.
I’ve also started using it recently and I’m not sure if the way I’m doing it is particularly “right”.
I don’t have a lot of knowledge of practical coding practices because in school we literally had a new project every two weeks so I never learned things like you need unit tests or proper architectural design. It was mostly making sure whatever project there was that week ran and didn’t crash.
So now I’m working as a sysadmin doing the random junk a sysadmin gets pushed on them. What I’ve been doing is telling it my project plan, Claude will write up something that looks better, and I continue to have a back and forth about architecture and libraries, asking it if it thinks any particular idea is good or bad, until I get to a place I’m happy.
Then because I want to learn rust and implement it myself, I’m having Claude basically guide me through creating it like a teacher would, with it taking on a very Socratic tone (“now that we’ve done this, what do you think is the next step?” “We have a list of CSVs so what do you need to do to read their values?”). And I’ve been moving forward but by bit like this.
I don’t know if it’s a particularly good way, honestly, I’d love feedback from anyone who’s done something similar or whatever!
I think it’s pretty heavily dependent on what you’re trying to do. I’ve gotten a lot of push from higher ups at my company to use copilot wherever possible. So, I’ve spent a lot of time lately having copilot + opus write code for me. Most of what I’m doing is super straightforward middleware APIs or basic internal front ends. Since it has access to very similar codebases for reference, and we have custom agents that point it in the right direction, it’s a pretty good experience.
However, if I ask it to do something totally new, it does okay, more like what you’ve experienced. It takes a lot of hand holding, but it usually gets the job done as long as you’re very descriptive in your prompt. Probably not faster than an experienced developer at the moment though
I’ve had an opposite experience. Here are some guidelines I follow:
- Setup a foundation of rules and knowledge for Claude to fall back on. I define expectations, common definitions, behaviors and anything else that’s not project specific upfront.
- in Claude.md I reference different domains of behavior, definitions, and rules (Claude has conventions for storing this type of stuff, so ask it to handle organizing information too)
- create a top-level project definition: this defines what “knowledge” is. It allows you to build up what Claude knows later on as you work on your project. “Update knowledge”, “add this to your knowledge”, etc
- create a top-level rule: all information in knowledge must have one source of truth. Whenever needed reference the original knowledge source instead of duplicating it. Now you can ask it to “review your knowledge”, “audit and flag knowledge”
- explicitly explain everything, leave nothing ambiguous; explain like you’re explaining the problem to a new developer who’s not familiar with the plan or codebase at all. Don’t ask it to write code right away. Ask it to write a plan/spec. Review the plan, make changes and discuss it until the plan is 100%. This plan can include implementation details if you’re ok with that, but it’s not necessary (sometimes I write a separate referenced file called implementation.md beside the plan and have the plan reference it.).
- Your role as a developer is shifting from writing code, to writing specs, and reviewing code
- Once there is nothing left to describe, and no ambiguity in your plan, have it use your plan to write the code. This works amazingly well for me.
A benefit to this method is that there is less wasted effort on my part. If Claude writes the code wrong, I can trace the reason for the mistake to a gap in the plan. I can then update the plan, throw away the code (if I have to), and have Claude reimplement the code again.
Rinse and Repeat.
Keep knowledge, plans, and implementation details clearly separated (you can copy your latest successful knowledge files to new projects to get started on future projects even faster).
Keep the goals of each plan as small and granular as possible (easier the define plans). Knowledge, plans, and implementation details all get tracked in your repository just like your code does.
I’m a career developer, and have been writing code for over 20 years. I’m adding this bit because I understand how AI driven development can look like a threat to developers. Over this last year, I’ve had a shift in this thinking though. I can take what I’ve learned through my career and use it to inform writing successful specifications Claude can use to write effective code. Claude may not solve all of our coding problems, but if used effectively, it solves nearly everything you throw at it.
hey shaggy. I want to touch on your last point as a newer developer:
My department is finally seeing 10x development due to the shift of writing code to writing spec. The main issue is now our pipeline is stuck at review, so all that extra output is effectively wasted. Do you have any tips on what worked for you if you had a similar situation?
This is our new bottleneck too. Developers roles are shifting to spec writers and code reviewers more and more. I don’t think I’d call this wasted effort though (unless the code produced is worse than what developers would have produced otherwise). I’d think of it as a good problem to have.
We’re doing several things to alleviate this, and I’m genuinely curious how other teams are handling this too.
- We have Claude running code reviews on our PRs too 😄. In our department, a PR isn’t expected to be reviewed by a dev until the author has addressed or reviewed and dismissed all of the issues Claude has brought up.
- There is pressure for developers on our team to become better reviewers. I think this is good, because reviewing code is a more valuable skill to prospective employers than writing it is anyway.
If you’re stuck at review you aren’t seeing 10x development, you’re seeing 10x code generation.
This is especially important because without the review/test/deploy part of the pipeline you aren’t actually seeing any progress towards business goals.
Once you do get these parts sorted, you can then look at what multiplier you’re seeing.
That’s not to say there isn’t an improvement in your workflow, just that you can’t say with any certainty what kind of improvement without measuring the end to end.
It might turn out that the rest of the pipeline is way easier , in which case your multiplier will be higher, it might also be much harder, in which case the multiplier will be lower.
I’m not taking shots, i mean it seriously, especially if you need to report any of this to the rest of the business.
edit : In addition, if it turns out that review is going to be a bottleneck you can get extra resource pointed in that direction which will benefit the workflow overall.
another edit: i would consider correctly managing the expectations of those you report to as a vital skill.
Exactly this. My experience with our companies wrapper on Claude lines up with OP, not this comment thread.
Everyone seems to forget everything you write is a liability. You can’t have bugs in code that is never written or generated, comments that don’t exist never become inaccurate, not duplicating “knowledge” into a repo doesnt have a risk of not aligning with business goals long term as they change.
From what I’ve seen, people claiming a “10x increase” did not have a strong foundation to begin with and/or did not utilize tools like IDEs effectively. No offense to thread OP, which seems itself a generated response, but in the time he has done all of that a strong engineer would be long done. Everything listed should be done before ever getting into code along with business and product partners.
I think it’s mostly going to be useful for boilerplate generation, and effectiveness is going to vary wildly based on what language you’re using. JS or Python? It’ll probably do OK. Plenty of open source for it to “learn” from. Delphi? Forget it.
Brief experimentation showed it liked to bullshit if it was wrong, rather than fix things.
Recently I used it (some free-tier DuckAI model, not Claude) to write a Python script for pasting PNGs into PDFs (complete with Tk interface) while applying a whole bunch of custom transformations. Simple enough, but a total chore with all the back-and-forth of searching for relevant unfamiliar libraries and syntax checking and troubleshooting. Inevitably it would have taken me the whole afternoon by hand. With AI I knocked it out in 25 minutes. That was my epiphany moment.
Since then I’ve noticed a general problem with AI coding. It almost always introduces too much complexity, which I then have to waste time untangling (and often just understanding) before I can proceed. Whereas if I had done it “my way” from the start I might have got there earlier. But I figure this problem is kinda on me.
And for me, therein lies why my use of it has become reduced to a really complex rubber duck, or to write something out that I could do by hand, but making my robot butler do it is just faster. Anyone actually leaning into today’s generative AI models for generating code that requires complexity or thought… they shall reap what they sow in the years to come.
You need to use plan mode
