Have you ever wanted to teach a robot to write poems or stories? Sounds impossible, but it’s easier to do than you might think! I recently read Max Woolf’s article, How to Quickly Train a Text-Generating Neural Network for Free, and worked through his Google Colaboratory notebook for a school project. It uses machine learning to train a Recursive Neural Network (RNN) how to write.
What I did:
Using Max Woolf’s pre-made code, I taught a bot to spit out text. You put in a file, run the code, and it starts learning from this text and spitting out text. It was originally trained on Shakespeare, but I used a .txt file of Alice in Wonderland by Lewis Carroll, downloaded from Project Gutenberg.
The computer is supposed to learn from the file you put in. There are lots of different settings, such as making it learn by word or by character, and I adjusted the settings a few times. My experiment was training different models of the Wonderland-trained bot by trying different combinations of settings. The results were interesting!
How I did it:
We found a pre-made code here at https://minimaxir.com/2018/05/text-neural-networks/ and opened it with Google Colaboratory. Next, I found the Alice in Wonderland file and uploaded it to the Colaboratory notebook. I used a couple different settings for each one, like changing the word level (whether it trains on all characters or on all words).
I trained two completely different models by changing the settings for the training epochs, meaning how many times the program went through the whole text file; the training size; dropout rate; and word level.
I changed the settings, and now I have 2 different models. It was really easy, and I just let it sit for a while so I didn’t have to watch it learn. When this was all finished, I copied and pasted the stuff it spit out into a Word Document.
Model 1 trained for 100 epochs at the word level, and was not very successful.
Model 2 trained for twice as long as model 2 and learned at the character level instead of the word level. It was much more successful, and sometimes even sounded like Alice in Wonderland!
What happened while I was doing it:
While I was doing it, it was slowly learning from the files. The first model had different settings, and it was pretty much just spitting out little bits of text and A LOT of quotation marks.
Example:
“
” ” ” “
, ll alice roots dreadfully i , , off off , dark had had alice had alice alice dark
comfort oh oh oh . to ! – i i – i ; ; are ! to that hour first ; ; i i i i i i ! ! i _ the
both other night see
_ see one baby hardly ” it one , it the baby more
The first model was starting to make sense near the end, which was good. The second model had just been starting to spit out sensible text near the end of its last epochs. I only had 100 epochs for the second model, and I had 200 for the first one, so I guess that gave it more time to learn.
What the results were:
The results for the first model were actually starting to make more sense, because it was finally starting to learn how the file had been working and doing that stuff. For the second model, the results weren’t really making sense, because it had just learned how to actually spit out text, and it still had weird commas, periods and quotation marks in between words.
Model #1:
Epochs: 200
Gen each epoch: 20
Word level: False
Train size: 0.8
Drop out: 0.0
File: Alice in Wonderland
200th epoch sample:
Model #1’s sample:
This is a sample from the 200th epoch of Model 1, which was the last epoch. I used the Alice in Wonderland .txt file. I had it generating a sample every 20 epochs, meaning it spit out text every 20 out of the 200 epochs. I also had it training at the character-level rather than the word-level. It didn’t have any dropout, and I set its training size to 0.8, meaning it had been training to be more like the file.
####################
Temperature: 0.2
####################
s very tired of stick, the poor little thing was saying, and in a sorrund at the door, she would feel
with bring fite their usual off, if you provide a free at encust acceptanted to have no
meaning to it, but she the root fell on it,
And with a great id yet.”
“I don’t know of it out it
This is temperature number 0.2, meaning that it is more predictable. It is also just a small sample of what I did here. I feel like this was making sense in some parts, but not making sense at all in others. You can see it was still making up words, like “sorrund ”, and it was still using a few random underscores every now and then. Still, there are parts that really create a sense of narrative, like “s very tired of stick, the poor little thing was saying, and in a sorrund at the door”.
####################
Temperature: 0.5
####################
y. “Oh, I begin–out-I forgoise—‘”
“Now, whater!” said the Queen, in a shill, the salt were places in his any looked at processions hands,” said Alice a little ask the find of the house of the house pig, that she was walking her face–and she is sounded out of the reedsact she sat still and shut
This is a sample at temperature 0.5, meaning it was halfway between really random and really predictable. I actually liked this part of the sample the best. It was still making up words, but it was really cool, because it was making up it’s own story, in a way.
####################
Temperature: 1.0
####################
“Ah! t
not even getting her tells of her choies,” thought Alice, “and if it seems was not almorded, “and take this could not.
“Yes, please do!” Alice replied in an wonderful, and she dreamed
on the large of the party deal),
How neatler, or and pebled slire, who all can be a very different pers
This is a sample at temperature 1.0, meaning it was very random. Again, still making up words, but I think with just a little more training this would’ve ALL been making sense.
That was the end of the sample for model #1, but now you can see the sample for model #2, for which I trained differently with different settings!
Model #2:
Epochs: 100
Gen each epoch: 10
Word level: True
Train size: 0.5
Drop out: 0.2
File: Alice in Wonderland
100th epoch sample:
I trained this model for only 100 epochs in total, with it spitting out samples every 10 of the 100 epochs. I used the same .txt file of Alice in Wonderland, but I changed up the settings a little bit. You can already see I changed the number of training epochs and word gen for the epochs, and I also changed the word level to True, meaning it was training off of word instead of character. I decreased the training size by 0.3, so that it was 0.5, and added 0.2 dropout, meaning it would drop out 0.2 of the text in the file to train on.
####################
Temperature: 0.2
####################
a to , . and know was it it she they – – – – – – , : only ” fancy fancy ” sudden
close ” ,
what ii be very , be be , very believe into it a t out it much much others c was
in in she had was day
a a bill stick , . you ‘ ‘
said a and a
large the the foundation – foundation a of ) of the the the hall it , it moment delight you you queer being
Okay, so, this is temperature 0.2 for my second model, and this is clearly A LOT different from my first model. I was still using the same file, but I changed the settings. It has a lot of weird spacing and a lot of quotation marks, commas, periods, etc.. I think it’s almost funny though!
####################
Temperature: 0.5
####################
, in alice , sha all , ! ! which this ! which quite comfort – to – goes goes – – with together it the – with ! ! ! ‘ ‘ the hadn _ i i as to have the – i i ‘ ‘ , if , _ _ was when ‘
, queen sighed was was t i ‘ and had
” and to illustration door door a to , your the and large ” been come
a to
frightened were ” will – will stand i because . go seemed it up a tied can can do cause she ‘ as one watch queen all all all their s course grammar of – – – – – of as – – –
, ” “
– and –
to the , ,
Ok, this was temperature 0.5 for my second model, and it was all over the place. It was the longest temperature out of them all, and it had a lot of weird spacing and quotation marks. I think with a little more training it could get the hang of it.
####################
Temperature: 1.0
####################
all you make doth ” was of put it the ” ground baby maps her ” ” into through m she plan might
he a creatures so
and i and to
a hedgehogs looking saying
, to of like left a
a asking the , had it
generally sensation found
change cat splash dormouse drew your to . ! ” and queen conversations great , the blasts , t t advice in in first first at matter first much ” the in that “
in witness in in in in and and in in who girls time she she seemed ) : one you – ve her found trial were about a ” – one witness ,
the witness – the about his i and out the is rustled
This was temperature 1.0 for my 2nd model. For some reason it likes to say “onions” a lot! I think this needed more time to train so it could get the hang of the spacing, but I think this makes it look cool and unique.
So those were all the samples to my 2 models! I hope you liked them!
Sisters 4 ever 