What to do BEFORE you start on your side-project
Sometimes, you just get an awesome idea for a project, whether for self-improvement or for profit. But what comes next is the desperation — you just don’t know where to start. And even if you do — you get lost after a couple lines of code.
So, what do you do?
The most important thing is to DIVIDE your project into smaller parts. This is the core of any sort of problem-solving, which is the literal definition of programming. A huge project that seems just impossible to complete on your own becomes a series of simpler, solvable tasks.
In order to do that, however, you have to define what your project is designed to accomplish.
I will give you an approximate plan of how to prepare for your project, and give you examples of how my thought process went with my own project:
1. What do I want to accomplish with my project?
I want to make it easier for new bug bounty hunters to discover new targets as quickly as possible by sending them notifications about new programs and recent changes in old ones.
2. What do I need to make that happen?
First of all, I need to scrape web-pages that contain information about bug bounty programs.
Next, I need to record the information that I obtain so that it is possible to compare new data to old data.
After that, I have to either detect changes in old programs or record new programs.
Then, I have to send users notifications if I detect changes or new programs.
3. How do I do that?
For web-scraping, I’ll have to make a python script that gets the text on the Hackerone pages and parse that text for the necessary data (which is where REGULAR EXPRESSIONS come in hand. Ugh)
For recording and retrieving information, I’ll need a database — my choice is PostgreSQL, which is powerful, well-documented and free
For detecting changes and new programs, a simple text comparison script is all I need
For notifications, I’ll need a telegram bot, which I’ll implement via a python package that is appropriately named ‘telegram’
Finally, in order to stay classy and make the whole application easier to deploy, I’ll need Docker
4. What problems will I face?
Aside from the regular coding, here are a few examples of challenges that come to mind
a) Optimization. Python is not the fastest language by itself, so I’ll need to make the other parts work smoothly. In my case, database queries are a problem — cutting unnecessary queries would be really nice. That can be done by storing the most important information in local variables instead of getting it from the database every time.
b) Hackerone is a Single Page Application. SPAs are notorious for being hard to scrape data from. In order to actually do anything, I’ll need to emulate a real user accessing the Hackerone pages. That is because a Single-page application needs to have it’s JavaScript executed in order for the contents of a page to render which is only possible in a browser environment, so a simple HTTP request won’t do). For Python-based SPA scraping, pyppeteer package is THE choice — it emulates a Chromium browser without the user interface which you can control using python
c) Parsing. That is a HUGE pain in any sort of web-scraping, because the data that you need is not standardized in any consistent way. For selecting only the data that you need, you’ll have to come up with a clever REGULAR EXPRESSION. Which hurts. Still, no matter how well you set the REGULAR EXPRESSION up, you’ll lose some data that does not adhere to the website’s format — you’ll just have to accept that.
When you have little experience in software development, just as I do, it is hard to foresee all of the challenges that will arise. Problems will come up and you’ll be stunned at how complex even a simple problem may be. Do not give up, however. Every problem solved will make you a better developer. Eventually, planning will become easy, and with that, any problem will be easier for you to solve.
5. How do I advertise my project?
This point is important even if you are not planning on turning your project into a product for a wide audience. Every single one of your projects is proof that you are a capable developer — don’t let them simply disappear in your filesystem.
The simplest thing to do is to upload your project to GitHub — do that with EVERY piece of code that you write. You’ll develop a portfolio that will impress your potential employers.
After that, you’ll have to put links to your GitHub account everywhere. Advertise yourself with your portfolio.
If you do plan on making your project used by other people, one of the best things to do is to be a part of a software developer community, or any community that your application might be useful for. Anything will do. I’ve picked reddit and decided to post there about my project. People will also give you feedback on your project, which is always good.
So, lets sum it up:
To successfully develop anything, you’ll have to break down your project into smaller problems. To do that, simply ask yourself — what do I want to accomplish and what will I need to do that? After that, you will have to think: “HOW do I do that?” This will help you pick all of the necessary technologies and packages for your application. Finally, you will need to think about all the problems that you will encounter on your journey — don’t worry, this part gets easier with experience, and it is absolutely necessary. Also, please think about advertising your project — your ideas deserve to be seen and appreciated.
Ta ta for now!
(my github is https://github.com/webMichaelNosenko/ for anyone interested in my project)
