Have you ever wanted to teach a robot to write poems or stories? Sounds impossible, but it’s easier to do than you might think! I recently read Max Woolf’s article, How to Quickly Train a Text-Generating Neural Network for Free, and worked through his Google Colaboratory notebook for a school project. It uses machine learning to train a Recursive Neural Network (RNN) how to write.
What I did:
Using Max Woolf’s
pre-made code, I taught a bot to spit out text. You put in a file, run the
code, and it starts learning from this text and spitting out text. It was
originally trained on Shakespeare, but I used a .txt file of Alice in Wonderland by Lewis Carroll,
downloaded from Project Gutenberg.
The computer is supposed to learn from the file you put in. There are lots of different settings, such as making it learn by word or by character, and I adjusted the settings a few times. My experiment was training different models of the Wonderland-trained bot by trying different combinations of settings. The results were interesting!
How I did it:
We found a pre-made code here at https://minimaxir.com/2018/05/text-neural-networks/ and opened it with Google Colaboratory. Next, I found the Alice in Wonderland file and uploaded it to the Colaboratory notebook. I used a couple different settings for each one, like changing the word level (whether it trains on all characters or on all words).
I trained two completely different models by changing the
settings for the training epochs, meaning how many times the program went
through the whole text file; the training size; dropout rate; and word level.
I changed the settings, and now I have 2 different models. It
was really easy, and I just let it sit for a while so I didn’t have to watch it
learn. When this was all finished, I copied and pasted the stuff it spit out
into a Word Document.
Model 1 trained for 100 epochs at the word level, and was not
very successful.
Model 2 trained for twice as long as model 2 and learned at
the character level instead of the word level. It was much more successful, and
sometimes even sounded like Alice in Wonderland!
What happened while I
was doing it:
While I was doing it, it was slowly learning from the files.
The first model had different settings, and it was pretty much just spitting
out little bits of text and A LOT of quotation marks.
Example:
“
” ”
” “
, ll alice roots dreadfully i , , off off ,
dark had had alice had alice alice dark
comfort oh oh
oh . to ! – i i – i ; ; are ! to that hour first ; ; i i i i i i ! ! i _ the
both other
night see
_ see one baby
hardly ” it one , it the baby more
The first model was
starting to make sense near the end, which was good. The second model had just
been starting to spit out sensible text near the end of its last epochs. I only
had 100 epochs for the second model, and I had 200 for the first one, so I
guess that gave it more time to learn.
What the results were:
The results for the first model were actually starting to
make more sense, because it was finally starting to learn how the file had been
working and doing that stuff. For the second model, the results weren’t really
making sense, because it had just learned how to actually spit out text, and it
still had weird commas, periods and quotation marks in between words.
Model #1:
Epochs: 200
Gen each
epoch: 20
Word level:
False
Train size:
0.8
Drop out: 0.0
File: Alice in
Wonderland
200th
epoch sample:
Model #1’s
sample:
This is a
sample from the 200th epoch of Model 1, which was the last epoch. I
used the Alice in Wonderland .txt file. I had it generating a sample every 20
epochs, meaning it spit out text every 20 out of the 200 epochs. I also had it
training at the character-level rather than the word-level. It didn’t have any
dropout, and I set its training size to 0.8, meaning it had been training to be
more like the file.
####################
Temperature: 0.2
####################
s very tired of
stick, the poor little thing was saying, and in a sorrund at the door, she
would feel
with bring fite
their usual off, if you provide a free at encust acceptanted to have no
meaning to it, but
she the root fell on it,
And with a great id yet.”
“I don’t know
of it out it
This is temperature
number 0.2, meaning that it is more predictable. It is also just a small sample
of what I did here. I feel like this was making sense in some parts, but not
making sense at all in others. You can see it was still making up words, like
“sorrund ”, and it was still using a few random underscores every now and then.
Still, there are parts that really create a sense of narrative, like “s very
tired of stick, the poor little thing was saying, and in a sorrund at the door”.
####################
Temperature: 0.5
####################
y. “Oh, I
begin–out-I forgoise—‘”
“Now,
whater!” said the Queen, in a shill, the salt were places in his any
looked at processions hands,” said Alice a little ask the find of the
house of the house pig, that she was walking her face–and she is sounded out
of the reedsact she sat still and shut
This is a sample at
temperature 0.5, meaning it was halfway between really random and really
predictable. I actually liked this part of the sample the best. It was still
making up words, but it was really cool, because it was making up it’s own
story, in a way.
####################
Temperature: 1.0
####################
“Ah! t
not even getting
her tells of her choies,” thought Alice, “and if it seems was not
almorded, “and take this could not.
“Yes, please
do!” Alice replied in an wonderful, and she dreamed
on the large of the
party deal),
How neatler, or and pebled slire,
who all can be a very different pers
This is a sample at
temperature 1.0, meaning it was very random. Again, still making up words, but
I think with just a little more training this would’ve ALL been making sense.
That was the end of
the sample for model #1, but now you can see the sample for model #2, for which
I trained differently with different settings!
Model #2:
Epochs: 100
Gen each
epoch: 10
Word level:
True
Train size:
0.5
Drop out: 0.2
File: Alice in
Wonderland
100th
epoch sample:
I trained this
model for only 100 epochs in total, with it spitting out samples every 10 of
the 100 epochs. I used the same .txt file of Alice in Wonderland, but I changed
up the settings a little bit. You can already see I changed the number of
training epochs and word gen for the epochs, and I also changed the word level
to True, meaning it was training off of word instead of character. I decreased
the training size by 0.3, so that it was 0.5, and added 0.2 dropout, meaning it
would drop out 0.2 of the text in the file to train on.
####################
Temperature: 0.2
####################
a to , . and
know was it it she they – – – – – – , : only ” fancy fancy ” sudden
close ” ,
what ii be very , be be , very believe into it
a t out it much much others c was
in in she had
was day
a a bill stick
, . you ‘ ‘
said a and a
large the the
foundation – foundation a of ) of the the the hall it , it moment delight you
you queer being
Okay, so, this
is temperature 0.2 for my second model, and this is clearly A LOT different
from my first model. I was still using the same file, but I changed the
settings. It has a lot of weird spacing and a lot of quotation marks, commas,
periods, etc.. I think it’s almost funny though!
####################
Temperature:
0.5
####################
, in alice ,
sha all , ! ! which this ! which quite comfort – to – goes goes – – with
together it the – with ! ! ! ‘ ‘ the hadn _ i i as to have the – i i ‘ ‘ , if ,
_ _ was when ‘
, queen sighed
was was t i ‘ and had
” and to illustration door door a to ,
your the and large ” been come
a to
frightened were ” will – will stand i
because . go seemed it up a tied can can do cause she ‘ as one watch queen all
all all their s course grammar of – – – – – of as – – –
, ” “
– and –
to the , ,
Ok, this was
temperature 0.5 for my second model, and it was all over the place. It was the
longest temperature out of them all, and it had a lot of weird spacing and
quotation marks. I think with a little more training it could get the hang of
it.
####################
Temperature:
1.0
####################
all you make doth ” was of put it the
” ground baby maps her ” ” into through m she plan might
he a creatures
so
and i and to
a hedgehogs
looking saying
, to of like
left a
a asking the ,
had it
generally
sensation found
change cat
splash dormouse drew your to . ! ” and queen conversations great , the
blasts , t t advice in in first first at matter first much ” the in that
“
in witness in
in in in and and in in who girls time she she seemed ) : one you – ve her found
trial were about a ” – one witness ,
the witness –
the about his i and out the is rustled
This was
temperature 1.0 for my 2nd model. For some reason it likes to say
“onions” a lot! I think this needed more time to train so it could get the hang
of the spacing, but I think this makes it look cool and unique.
So those were
all the samples to my 2 models! I hope you liked them!
Sisters 4 ever 