Programming. How hard it can be? Part 2

How to build familiarity with a subject that we know nothing about? This is a million dollar question. I realize the difficulty that we are facing. Different people have different learning styles. For some of us classroom setting might be the right one. Others will rely on some tutorials and visual help. There is also tendency to learn things the right way, whatever that means. It is logical to start with introducing some simple concepts like loops and eventually move on to objects, methods and classes. It does make sense if you are looking from that perspective.

Most of us are guilty of believing that what works for us must be also good for general public. I will advise some caution on that matter. Sometimes, I have my doubts too, whether I’m learning things in a correct way and if I actually wasted some valuable time. There is always some risk associated with it.

I have to admit that I‘m not a big fan of step by step tutorials. Neither I like exercises and programming challenges. I’m sure they are valuable tools. However I would like you to consider couple things before you proceed on your coding venture. Think about what software application is and how it works. It could be desktop, web application or a phone app. You take information from somewhere in form of a file. You store that information in memory or in a database. You perform some manipulation on that data and finally you display it to the end user. You may also take some input, validate it and pass it somewhere else. This is rather general description but it is actually true for most scenarios.

Does learning all these necessary steps seem like a proper challenge? Considering that you might be learning programming on your own and you have limited time to spare, would you devote most of it to do some online exercises or concentrate on things that actually make up elements of an application?

As for doing it the right way, I would strongly advise taking different approach. You probably thinking that in order to do something you need to have enough knowledge. Yes and no. Of course it is better when you expand on subject as much as possible. On the other hand should you wait until you learn everything that is possible on that topic? Let me give some example that is close to learning programming and that is learning foreign language. Imagine that for some reason you find yourself traveling in a foreign land and your vocabulary is very limited. Would you try to communicate your needs with words that you already know, or you would not talk at all because your grammar is incorrect? Maybe you have to build more complex sentences with few words that you are familiar with. After all you are not going to wait until you will get it right. That would seem like a wasted opportunity.

My findings for taking programming skills to the next level prove to be on opposite site to what I just described. So, there is no one rule fits all approach. I will write about in different article.

You are probably thinking what kind a nut job is telling you to disregard all rules, avoid tutorials and build functioning application from scratch when you cannot even write simple method. Yeah, that’s pretty much what I would like you to do. In fact, I would make this mandatory. Fire up Google and ask some questions. Figure out what elements do you need to scan folder. Get file names and paths and show them in some list.

If you serious about programming, you can absolutely do it by yourself.

 

To be continued…

Programming. How hard it can be? Part 1

Couple years ago, I picked up a book about Java with a clear intent of learning programming. After reading “Hello World” chapter and some other paragraphs, I put it back on a shelve and never looked back. There was nothing wrong with that book. It was written by highly respectable authors and it also had very good reviews on Amazon. Yet, the learning curve was too steep at that moment. Instead of giving up completely, I read two other books about VBA programming for Microsoft Excel and Access. As required, I did all necessary exercises and eventually after completing it I did not retain much of that information.

The choice of the language itself was not premeditated. At that time, Java seemed to me like a good starting point. I did not know too much about different languages, and since there are so many of them, the choice I had to make was rather accidental.

Now, if you asking yourself a question where should I start or what programming language I should learn, then this article series may provide some clarity on the subject. My intention is not to give you ready solution, but instead explain some of the things that may seem not that obvious to somebody who wants to start on a path of becoming programmer or developer.

Everybody says that learning programming is about passion and writing code. Although it’s true to some extent, it goes beyond that. It is more about method that you use and rationale behind it. Authors of a book may have a good intentions putting “Hello World” and other paragraphs about loops and if statements, but at the end you as reader feel like you hit a brick wall. Also, ask yourself a question how many of these authors are actually self-taught programmers? Some of them probably are, but in most cases they have a good professional background with Computer Science degree. Now imagine if I asked you do you want to be an Architect or a Lawyer and I gave you some books to study. Then I will tell you to do all exercises and promise to return to check your progress in six months. Does this seem like something realistic? Probably not, yet many aspiring programmers and developers fall into this trap when they subscribe to this way of thinking.

Choice of the language is not that obvious either. Type in Google or YouTube question what language I should learn and you will get tons of answers and opinions. Some people will recommend starting with Python; others will direct you straight to C++. Everybody has some rationale behind it and they would give you a list of associated benefits. Python is considered as one of the friendliest languages to learn. C++ has an opinion of one of the hardest to digest. So, if someone is telling you to start with C++ because after you learn this language everything else would seem easy. I would add another statement that you need to learn how to write a novel. Once you master it, you will definitely know how to write an essay too.

Now let’s go back to my story from the beginning. I picked up that Java book thinking how hard it can be to learn programming? Apparently, there is more to it than just reading some material or even writing some code. I will explain this in upcoming articles. It also made me think why I failed on my first attempt and why I was somewhat successful plowing through VBA material. It’s all about familiarity with subject at hand. I had some experience working with VBA and databases at my workplace. I was completely at home and I was only expanding on my knowledge. Java on the other hand was a terra incognita that quickly closed the gates and kept me at bay while I was hoping to get inside one day.

Roadmap

I started working on converter at the beginning of last summer. I was not concerned with the code itself but rather I wanted to get something that would do the job. I created other tools before, but as you can already guess, I did them mostly to automate some specific aspect of my HTPC setup.

I had a need to move Plex database from my server to another computer. At first it looked like difficult task, but then I realized that I had some pieces of the code already written. Thirty minutes later, I had a working solution and functioning Plex server running on NUC.  It hit me that I should get my programming skills to the next level.  Over the next couple months, through trial and error, I finally was able to get on the right track. I will write series of articles about that process that might be beneficial to some of you.

The converter was meant to be a quick fix to one problem. As I described previously, I was not satisfied with other tools and that’s why I started to working on this one. Unfortunately, the source code is very messy and it is getting to the point when implementing new features is near impossible. Not to mention, that creating different user interfaces would pose some challenges. Therefore, beta 0.4 will most likely be the last iteration of this app in this form.

In the future I would like to have web based app that has exactly the same functionality. That means I would like to be able to access application’s web interface on another computer in my house. I would like view available recordings and add them to queue. The good news is that I have pretty good idea about the components that this new app has to include. I do not have to start everything from scratch. Although, transferring some ideas into Object Oriented Programming proves to be sometimes difficult. Again, I will write some articles about it too.

On HTPC front, I tested DVR capabilities of yet to be released beta version of Plex server and to tell the truth I have some mixed feelings about it. Initially, I wanted to restrict my blog to programming, but on the second thought I will include this type of topics too.

Stay tuned.

 

New Feature: Queue up movies from a channel

 

This release contains one important fix and a new feature. Some of the users reported that GUI was crashing during startup showing error that date tag could not be properly converted. Beta 0.4 should take care of it once for all. In addition there is a new feature that will allow you queue up recordings from a specific channel that shows many movies. What you have to do is to select name of the channel and profile you would like to use. Then save it. It should show up in the right panel. Any modifications you would like to make later, would require deleting that channel from a list and save it again.

Once you select the channels you want you can either run app from a Task Scheduler or go to Tools and select Build Queue.

Queue Channels is available under Tools.

Update: Fixed scanning speed of an input folder

This is the last update in this calendar year. It fixes the scanning speed of an input folder. Previous update introduced this issue.

I made some changes to the used method. Originally, we had to know location of a header id number for WTV file. These numbers differed from one operating system to another. Now, we take sample file, map it ourselves by using header name and pass it to the method that extracts metadata. This way we do not need to worry about what Windows version we are using. I tested application on Windows 7 and Windows 10. It should work on Windows 8 too.

Although scanning speed is really fast, you need to consider how many recordings you have in your Recorded TV folder. I have about 3.5 TB of data that takes about 2 minutes to scan over LAN.

Please let me know if you encounter any issues.

Beta 0.2 released

This updated attempts to fix following issues:

  • Added support for Windows 8
  • Preventing manual recordings from being processed
  • Preventing Windows Media Center sample files from being processed
  • Added logs for queue and encoder
  • Removed duplicate episode tag in nfo files
  • Filename tag in xml files was showing full name instead of filename without extension

Note. When updating this application, please remove all xml files from \mpconverter\metadata folder.

Initial scan of Input Folder takes longer than previously. It can actually take couple minutes. This is due to changed method and this is something that will be addressed in future release. I recommend monitoring activity in \metadata folder during initial scan.

“WindowsApplication1 has stopped working” error

Problem:  Application was installed correctly.  However, after editing settings and changing input folder location from “C:\Recorded TV” to “C\Users\Public\Libraries\Recorded TV” application is unable to start properly.

Cause: Some of the WTV files have their metadata tags empty.  It is most likely caused by Windows Media Center sample files or corrupted recordings.

Solution: Edit Folder Options. Select View Tab and check “show hidden, folders”.  Navigate to “C\Users\Public\Libraries\Recorded TV\Sample Media” and delete its content. Next go to “C:\Users\YourUserName\Appdata\Roaming\mcponverter\metadata” and delete its content too. Start application and let it scan Recorded TV folder.

Note: Sample WTV files do not have recording and broadcast dates. Application extracts metadata and creates xml files inside \metadata folder but when it starts again it parses xml files and it expects dates to be there. I will try to address this issue in next release. It is possible that your sample folder is already empty. In this case, some of the files may have similar issue.

This application was developed on Windows 7. I was able to test it on Windows 10.  It was not tested on Windows 8. Each Window version reads WTV metadata little bit differently. This is something you should be aware of.

 

mpconverter beta 0.1

 

mce_1

 

 

The reason I started working on this converter is that I wanted to bring over some recordings from Windows Media Center to other platforms like Kodi and Plex.  There are other tools that can do the job, however they rely on external services like thetvdb.com. Basically, these tools convert video file and fetch metadata from other websites. The problem is that some TV Series do not exist in thetvdb.com and they will never be allowed over there due to some licensing restrictions. I’m talking mostly about sports events or competitions. So what to do when you would like to archive some of your recordings and create necessary metadata? This is where mpconverter comes in.  You can easily convert entire series or sport event and add it your library.

My second concern was the quality of encoding. In a sense, I wanted to have greater control over this process. I tried out some other tools and I found it to be little bit too complicated and unpredictable. I do most of my conversions using Handbrake and its built in encoding profiles. Therefore I wanted to have this ability in my application. I also included FFMPEG as second option.   During the testing, I discovered that running FFMPEG in silent mode is more reliable than Handbrake. As a result FFMPEG became default encoder. However, if you would like to try Handbrake, you simply make that choice in settings. Just to clarify that we are using HandbrakeCLI. In the future I would like to use actual handbrake library

The tool is intended to be run from Task Scheduler . This allows to have encoding process to be done in a background. You can also start encoding process from Queue window. My tests revealed that there is very minimal impact on HTPC performance and the speed of encoding is almost 1:1. My HTPC is rather old system based on AMD quad core CPU and 4GB or RAM. Nothing spectacular,  but it is reliable. So if there is a 30 minutes recording, it would take about 35 minutes to complete a job.

I created GUI, but it is mostly to allow you setup your preferences and specify TV Series that you would like to be monitored.  As I said before, you should run encoding process from Task Scheduler.

 

mainform

 

 

Initial Setup

To begin you need to install mpconverter. During setup, it will create necessary folders, encoding profiles and it will also scan your TV Recordings folder. Depending on the size of that folder it can take 30 seconds or more. Be patient,  at the end you will see a list of your recordings in left panel.

Application assumes that your recording folder is stored on C drive like this “C:\Recorded TV”. If this is not a case head over to Settings and make necessary changes. You also need to set Output folder and Temp folder. I would recommend that you create this folders on hard drive that has plenty of free space. As you know WTV recordings take a lot of space and you need that space for temporary file.

TV Channels

Depending on your cable provider some of TV Channels may be copy protected. In this case you should prevent processing recordings that contain copy restriction flags. To do that, go to Tools and select Channels. Next uncheck channels that you would like be excluded.

Channels list is built during initial setup process and that information comes directly from your recorded files.  That list may vary. Every time queue builder adds new items to queue, it also inserts new channels to a list. Therefore you should review the list periodically to see if there are any new channels that require your action.

Season Pass

This is where you decide which TV Series you would like to be monitored. You simply select you favorite series from drop down menu and press add button. Now you  need to specify preferred encoding profile. Also, file type which can be either MKV or MP4. And finally you need to type in season name. It can be a standard season number like 1, 2 or 3. But it can also be year like 2016.

We do not use thetvdb.com database yet. Therefore, if you would like to process some series manually, you would also need to find out show season number and enter it the form.

Please do not leave any fields empty as omission may cause application to throw some errors.

Queue and Encoder

If you head over to folder

C:\Users\UserName\AppData\Roaming\mpconverter\bin

You will find two files. One is called mpconv_queue and second is called mpconv_encoder. They have distinct responsibilities. Mpconv_queue will scan your recorded TV folder and based on the criteria you specified in SeasonPass, it will match your request with any recording and it will put them into a queue. Mpconv_encoder will take it from here. It will read your queue file and initiate encoding process. Finally, when it’s completed, it will rename converted file and put it inside output folder.

Why there are two different files? I wanted to have ability to schedule queue at different times and not necessarily  start the encoding process immediately. So, you could create task in scheduler that only triggers queue activity or encoding activity. You can also create task that includes both activities run in sequence. And for the most of the users it may be the best approach.

 

 

kodi_1

 

 

Encoding Profiles  

You can find encoding profiles in following folder:

C:\Users\UserName\AppData\Roaming\mpconverter\profiles

There are profiles for both encoders. Handbrake profiles are the same as those that you can find in the desktop application. You will recognize their names. Getting profiles for FFMPEG is little bit more complicated. I was able to narrow it down to 2 standard profiles , one is to create 720p file and another one for Mobile device.

If you open profile xml file, you will find only 3 tags. One is for encoder name, second is for profile name and third is a command line input.  If you would like to make some modifications, all you have to do is to create new xml file. Give it some name and type in proper command line arguments. You do not need to include input and output file info.  Be advised, that FFMPEG may throw an exception if you make some mistakes.  So, use Google to search for samples.

 

img_3401 img_3402

 

 

 

 

 

 

 

 

 

 

 

 

 

Notifications

mpconverter includes ability to notify you about encoding status. This is done through email. You need to enter all necessary information in GUI application settings.

I do not collect any email information or any sensitive information. I have no plans of doing it now or in the future. Just to let you know, that your credentials are stored in xml global settings file. If you are worried about your security, simply do not use this feature

If you plan to use it, then you will receive email every time your run mpconv_queue. It will contain all the items that are currently in your queue, including the ones that were already completed.

mpconv_encoder will send emails when encoding process starts and when it is completed. So for every file, you should receive two emails. If for some reason you will not see a second one, it will mean that something went wrong.

 

 

img_3404 img_3406

 

 

 

 

 

 

 

 

 

 

 

 

 

Metadata

I mentioned before that major source of metadata are sites like thetvdb.com. Since, we do not use them at this moment, metadata comes directly from recorded file. That includes series and episodes titles; episode description; genre and recording time. For obvious reason we do not have TV Series description or fanart. This part it’s you own responsibility.

mpconverter creates standard nfo files for Kodi. So you should have no problems importing them into your library.

Naming convetion for indivudal file looks like this

\TV SeriesName\Season Number\TV SeriesName – S0XE0X – EpisodeName

If the series exists in thetvdb.com, then you should not have any problems importing it into Plex either. It will scan the folder and fetch necessary metadata.

For those recordings that do not have entry in thetvdb database, you will have to rely on the importer that was created by Plex community. It’s called XBMCInfoImporter and it will read Kodi nfo files and import them into Plex Library. There is simply no other way of doing it right now.

You can get XBMCInfoImporter from here

 As you probably know, Windows Media Center has its own set of issues with Guide Listings missing description or even episode titles. I would like to have ability to edit metadata directly in WTV file. I’m looking into this. At this point metadata is extracted from a file and put into a xml file that is located inside

C:\Users\UserName\AppData\Roaming\mpconverter\metadata

These files have the same name as original WTV files. If you cut out commercials in video editing software, save them and do not change their names. Otherwise new xml file will be created.

Some video editors may affect metadata stored inside WTV file. Warning! Leave original WTV file in Recorded TV folder, always work with a copy of a file.

Some recordings may not have a proper episode title. This is mostly true for some daily repeating shows. In this case we use following naming convention

\TV SeriesName\Season Number\TV SeriesName  (Day of the Week, Month, Day Number, Time Recorded)

Example: SeriesName (Sun, Oct 16, 22.55 PM)

Commercial skipping

mpconverter does not support commercial skipping. Over the years I tested various tools and I found process of identifying commercial breaks to be highly unreliable.  Every once a while comskip or ShowAnalyzer fails in the department. The same goes for VideoReDo. It turns out, that before you commit to any changes, the best way is to review cuts first.

So this is what I would recommend doing it. Use video editor and scan WTV file. Edit it and cut out commercials. Export new file to WTV and put it back into Recorded TV folder.  Make sure that file name is the same as before, otherwise mpconverter may encounter some issues with metadata.

Renaming Files

I you would like to rename WTV files and create necessary metadata, I added profile called RenameOnly. Original files will remain unedited, but they can be organized in folder structure that can be recognized by Plex and Kodi.

In addition, sometimes during conversion something may go wrong, so you will end up with completed MKV file inside temp folder. In this case you can still rename file and create Kodi metadata file by going to Tools and selecting Rename Files.

Task Scheduler

There are various options when you create new task in scheduler. In my own experience, I know that sometimes the same options selected on one PC, may not be replicated on another PC. In fact when you search internet, you will find posts where different users describe their own success and failure stories.

Task scheduler gives you an option of running task hidden. You may be asked to provide your credentials when you press OK button. Windows also gives you another option for scheduling hidden task. You can do this by changing user that creates task. Simply press “Change User or Group” and replace your default user name with “SYSTEM”. Next, reboot your PC. Try to run task and monitor temp folder to see if anything is happening.

I had more success running mpconverter with FFMPEG as main encoder, for some reason HandbrakeCLI refuses comply with request.

Under Actions tab select “New” and “Start a Program”. If you want to run both queue and encoder at the same time include them both, otherwise create two separate tasks.

 

 

plex_1

 

 

How does this work?

Let me explain what actually happens during the process. We start with mpconv_queue. It scan Recorded TV folder, extracts metadata for any new recordings. It creates necessary xml files and puts them in metadata folder. It also has some clean up duties. If WTV file was deleted, then any corresponding metadata file is also removed. In addition, any entry that exists in queue file that does not have original WTV file anymore, will be deleted too. Therefore you do not need to worry about your queue. It will be automatically cleaned up as soon you delete WTV file from your hard drive.

Conversion process is two-steps activity. First, we remux WTV to MPEG file using FFMPEG. We simply take video and audio stream, separate them and put them back together in different container. FFMPEG has ability to fix some time stamps issues. Please do not be worried about any error messages that FFMPEG shows in console window. This seems to be quiet normal. Mpeg file is created in Temp folder.

Second step, is actual encoding that can be done with either FFMPEG of Handbrake. Once it is completed, MKV or MP4 file that was created is renamed and moved to output folder. Finally queue file is updated.

What to do when something goes wrong?

Since the encoding process is done in a background you should have some idea how long it takes to convert one hour show on your HTPC. In this case email notifications are very helpful. If you do not receive second email telling you that job was completed, then you should start investigating. First thing you should do is to check Temp folder to see if there are any leftover files. If temp folder is empty, take a look in Output folder to see if converted file exists. Also check queue file to see if any of the items has “true” written in started tag. Clear that tag.

Check Task Scheduler to see if created task is still running. Stop it. Reboot your HTPC.

Sometimes it happens that FFMPEG and Handbrake hangs during mkv encoding. It is unclear why MPEG/WTV files fail during the process. In this case, reboot your PC and delete any remaining files from Temp folder. Re-Queue items and start conversion again. In most cases, files will be converted successfully.

If want to re-process the same file make sure that tags started and completed in queue file are empty.  Encoder keeps track of processed files in order to assign correct episode number. Go to following folder and check if episode xml file was created. You can safely delete last one, this way any new encoded file will have the same number.

C:\Users\UserName\AppData\Roaming\mpconverter\completed

Windows

I’m running this tool on Windows 7 64bit. I did not have a chance to test it on Windows 8.

Windows 10 will not play WTV files out of the box, however mpconverter will work with your recorded files. Simply point Recorded TV folder in Settings to your network share.

Final thoughts

I hope you will enjoy using this tool. I tried to make is as simple and straight forward as possible. I have it running on my HTPC for last couple months and it does really good job processing files that I want. I can setup recording and have it ready to sync to my phone Plex app before I leave for work. Also, when I get back home, I know that it will show up as TV Series in my Kodi and Plex library. I like the fact that I can specify which shows or episodes I want to process. In addition, I can decide at what time it should be done. That gives me greater control over conversion. Quality of final product is predictable too. So no more guessing how it will look too. I made over 100 conversions so far, and I did not encounter any major issues.

Although it is a beta release, you should give this tool a try. It is still under development. This is more like a proof of concept. I’m planning to make some changes to source code and interface will eventually change too.

If you encounter any issues or if you have any questions please post them in comments below. Thank you.

Enjoy it!

Hello everyone!

This is not your typical tech site or blog that brings daily news about current events. Occasionally I may bring my own opinions, however for the most part I will try to make this place interesting for everyone who likes technology and programming.

I have two major hobbies.  First one is HTPC and media. That includes everything that relates to recording, playing, storing media files. Second hobby of mine is programming that is done primarily in VB.net. I’m learning new programming languages, so this would be a good place to share my experience and findings.

I have couple projects that I’ve been working on. I would like bring them to a wider audience. They are still work in process, however they are functional and some of you may find them interesting enough to include them in your daily life.

My interest in programming is driven mostly by media consumption. Occasionally I find something in my media setup that annoys me and requires improvement or automation.  This is how ideas are born for a new program or script. I tinker and I try it until I find desired outcome. After a while when I use it, it becomes much more clear that certain things require different approach. As a result I return to a drawing board and make some changes to improve performance.

One of my projects that are under constant development is internet radio recorder. It started as simple vbs script that organized radio recordings and it moved to vb.net program that handles creation of RSS feeds. Next phase of development is actual internet radio stream recorder based on popular videolan player.

Second project is a video converter that would take WTV files created by Windows Media Center and encode them using FFMPEG or Handbrake to a smaller and more manageable video format like mkv or mp4.

My media setup consist of custom build HTPC computer, home server running unRaid, intel NUC computer that records radio streams, custom build home PC, Roku and iPhone. TV Recordings are done by Windows Media Center, but I prefer using Kodi for playing my movies and TV series collection. I also use Plex during my commuting hours.

I’m glad you could join me and stay tuned for next blog posts.