{"id":2502,"date":"2020-02-07T04:00:57","date_gmt":"2020-02-06T19:00:57","guid":{"rendered":"https:\/\/julialang.kr\/?p=2502"},"modified":"2020-02-07T04:01:00","modified_gmt":"2020-02-06T19:01:00","slug":"flux-cifar10-example-gpuminibatch-fix-loss-nan","status":"publish","type":"post","link":"https:\/\/julialang.kr\/?p=2502","title":{"rendered":"[Flux] cifar10 example &#8211; gpu,minibatch, fix loss NaN"},"content":{"rendered":"\n<p>model-zoo :  <a href=\"https:\/\/github.com\/FluxML\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10.jl\">https:\/\/github.com\/FluxML\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10.jl<\/a>  \uc18c\uc2a4\ub97c \uc218\uc815\ud568<\/p>\n\n\n\n<p>\uc218\uc815\ub41c \uc18c\uc2a4 :  <a href=\"https:\/\/github.com\/mrchaos\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10_gpu_minibatch.jl\">https:\/\/github.com\/mrchaos\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10_gpu_minibatch.jl<\/a> <\/p>\n\n\n\n<p>\uc774 \uae00\uc744 \ubcf4\uae30\uc804\uc5d0 \uba3c\uc800 \uc544\ub798 \uae00\uc744 \ucc38\uc870 \ud558\ub294 \uac83\uc774 \uc88b\ub2e4.<\/p>\n\n\n\n<figure class=\"wp-block-embed-wordpress wp-block-embed is-type-wp-embed is-provider-power-up\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"wp-embedded-content\" data-secret=\"YqOxAdtsnb\"><a href=\"https:\/\/julialang.kr\/?p=2495\">[Flux] mnist example with gpu, mini-batch, fix loss NaN<\/a><\/blockquote><iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;[Flux] mnist example with gpu, mini-batch, fix loss NaN&#8221; &#8212; Julia ?\" src=\"https:\/\/julialang.kr\/?p=2495&#038;embed=true#?secret=RvgslJHANP#?secret=YqOxAdtsnb\" data-secret=\"YqOxAdtsnb\" width=\"500\" height=\"282\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>\uadf8\ub9ac\uace0 cifar10.jl\uc744 \uc774\ud574\ud558\ub294\ub370 \ub3c4\uc6c0\uc774 \ub420\ub9cc\ud55c jupyter notebook(html) ==&gt; <a href=\"https:\/\/julialang.kr\/wp-content\/uploads\/2020\/02\/cifar10\uc758-\uc774\ud574.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"\uc5ec\uae30 (\uc0c8\ud0ed\uc73c\ub85c \uc5f4\uae30)\">\uc5ec\uae30<\/a><\/p>\n\n\n\n<p> cifar10_gpu_minibatch.jl   (\uba87\uac00\uc9c0 \uc124\uba85\uc774 \ub354 \ucd94\uac00 \ub418\uc5b4 \uc788\uc74c)  <\/p>\n\n\n\n<div class=\"wp-block-file\"><a href=\"https:\/\/julialang.kr\/wp-content\/uploads\/2020\/02\/cifar10_gpu_minibatch.zip\">cifar10_gpu_minibatch<\/a><a href=\"https:\/\/julialang.kr\/wp-content\/uploads\/2020\/02\/cifar10_gpu_minibatch.zip\" class=\"wp-block-file__button\" download>\ub2e4\uc6b4\ub85c\ub4dc<\/a><\/div>\n\n\n\n<p>\uc704\uc758 \ucca8\ubd80\ud30c\uc77c\uc5d0 \uc18c\uc2a4\uc5d0\uc11c &#8220;CUDAnative.device!(3)&#8221; \uc744 \uc8fc\uc11d \ucc98\ub9ac \ud558\uba74 \uccab\ubc88\uc9f8 GPU\uc5d0\uc11c \uc2e4\ud589\ub41c\ub2e4<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#=\n cifar10 dataset spec\n - 60,000 images of 32x32 size \n - train images : 50,000\n - test images : 10,000\n - classify item : 10\n - each class have 6,000 images and 5,000 train images, 1,000 test images\n \n Data format:\n WHCN order : (width, height, #channels, #batches)\n ex) A single 100x100 RGB image data format : 100x100x3x1\n =#\n\n# Julia version : 1.3.1\n# Flux version : v0.10.1\n__precompile__()\nmodule _CIFAR10\nusing Flux, Metalhead, Statistics\nusing Flux: onehotbatch, onecold, crossentropy, throttle\nusing Metalhead: trainimgs\nusing Images: channelview\nusing Statistics: mean\nusing Base.Iterators: partition\nusing CUDAnative\nusing CuArrays\nCuArrays.allowscalar(false)\n\nusing BSON: @save\n#using MicroLogging\nusing Logging\nusing Dates\n\nconst model_file = \".\/cifar10_vgg16_model.bson\"\nconst log_file =\".\/cifar10_vgg16.log\"\n# Very important : this prevent loss NaN\nconst \u03f5 = 1.0f-10\n\n# use 1nd GPU : default\n#CUDAnative.device!(0)\n# use 2nd GPU\n#CUDAnative.device!(1)\n\nlog = open(log_file, \"w+\")\nglobal_logger(ConsoleLogger(log))\n@info \"Start - $(now())\"\n@info \"Config VGG16, VGG19 models ...\"\nflush(log)\n# VGG16 and VGG19 models\n\nvgg16() = Chain(\n  Conv((3, 3), 3 => 64, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(64),\n  Conv((3, 3), 64 => 64, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(64),\n  MaxPool((2,2)),\n  Conv((3, 3), 64 => 128, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(128),\n  Conv((3, 3), 128 => 128, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(128),\n  MaxPool((2,2)),\n  Conv((3, 3), 128 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  Conv((3, 3), 256 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  Conv((3, 3), 256 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  MaxPool((2,2)),\n  Conv((3, 3), 256 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  MaxPool((2,2)),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  MaxPool((2,2)),\n  x -> reshape(x, :, size(x, 4)),\n  Dense(512, 4096, relu),\n  Dropout(0.5),\n  Dense(4096, 4096, relu),\n  Dropout(0.5),\n  Dense(4096, 10),\n  softmax) |> gpu\n\nvgg19() = Chain(\n  Conv((3, 3), 3 => 64, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(64),\n  Conv((3, 3), 64 => 64, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(64),\n  MaxPool((2,2)),\n  Conv((3, 3), 64 => 128, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(128),\n  Conv((3, 3), 128 => 128, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(128),\n  MaxPool((2,2)),\n  Conv((3, 3), 128 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  Conv((3, 3), 256 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  Conv((3, 3), 256 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(256),\n  Conv((3, 3), 256 => 256, relu, pad=(1, 1), stride=(1, 1)),\n  MaxPool((2,2)),\n  Conv((3, 3), 256 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  MaxPool((2,2)),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  BatchNorm(512),\n  Conv((3, 3), 512 => 512, relu, pad=(1, 1), stride=(1, 1)),\n  MaxPool((2,2)),\n  x -> reshape(x, :, size(x, 4)),\n  Dense(512, 4096, relu),\n  Dropout(0.5),\n  Dense(4096, 4096, relu),\n  Dropout(0.5),\n  Dense(4096, 10),\n  softmax) |> gpu\n\n# \n# Function to convert the RGB image to Float64 Arrays\n#=\n1)channelview\ub85c \uc774\ubbf8\uc9c0\uc758 color\ub97c channel\ubcc4\ub85c \ubd84\ub9ac\ud55c\ub2e4.\n- \ubd84\ub9ac\ub41c channel\uc740 \ub9e8\uc55e\uc5d0 \uc0c8\ub85c\uc6b4 \ucc28\uc6d0\uc744 \ucd94\uac00 \ud558\uc5ec channel\uc744 \ubd84\ub9ac\ud55c\ub2e4.\n- \uc608) 32x32 \uc774\ubbf8\uc9c0\uc758 \ucc44\ub110\uc744 \ubd84\ub9ac\ud558\uba74 3x32x32\ub85c 3\uac1c\uc758 \ucc44\ub110\uc774 \ucd94\uac00 \ub41c\ub2e4\n2)permutedims\ub85c \ubd84\ub9ac\ub41c \ucc44\ub110\uc744 \ub4a4\ub85c \ubcf4\ub0b8\ub2e4.\n- Flux\uc5d0\uc11c \uc0ac\uc6a9\ub418\ub294 \uc774\ubbf8\uc9c0 \ud3ec\ub9f7\uc740 WHCN-width,height,#channel,#batches \uc774\ub2e4\n- \ucc44\ub110\ubd84\ub9ac\ub41c \uc774\ubbf8\uc9c0\uac00 3x32x32\uc778 \uacbd\uc6b0 permutedims(img,(2,3,1))\uc744 \uc801\uc6a9\ud558\uba74\n- 32x32x3\uc73c\ub85c width,height,#channel \uc21c\uc73c\ub85c \ubc14\ub010\ub2e4.\n=#\ngetarray(X) = Float32.(permutedims(channelview(X), (2, 3, 1)))\n\n\n@info \"Data download and preparing ...\"\nfunction make_minibatch(imgs,labels,batch_size)\n  data_set = &#91;(cat(imgs&#91;i]..., dims = 4) |> gpu, \n          labels&#91;:,i]) |> gpu \n          for i in partition(1:length(imgs), batch_size)]\n  return data_set\nend\n# Fetching the train and validation data and getting them into proper shape\n#=\ntrainimgs(\ubaa8\ub4c8\uba85) : \n - \ubaa8\ub4c8\uba85\uc774 \ub4e4\uc5b4 \uac00\uba74 \ubaa8\ub4c8\uba85\uc5d0 \uad00\ub828\ub41c train\uc6a9 \ub370\uc774\ud130\ub97c \ub2e4\uc6b4\ubc1b\uc544 \ub9ac\ud134\ud55c\ub2e4.\n - ex) trainimgs(CIFAR10) : 50,000\uac1c\uc758 train data\uac00 return \ub41c\ub2e4.\nX \n=#\nepochs = 100\nbatch_size = 200\n\nX = trainimgs(CIFAR10)\n# Training\uc6a9 \ub370\uc774\ud130 \uc900\ube44\n# \uc774\ubbf8\uc9c0 \ucc44\ub110 \ubd84\ub9ac \ubc0f \uc7ac\ubc30\uc5f4, training\uc6a9\uc73c\ub85c 60,000\uac1c\uc911 50,000\uac1c\ub97c \uc0ac\uc6a9\ud55c\ub2e4.\ntrain_idxs = 1:49000\ntrain_imgs = &#91;getarray(X&#91;i].img) for i in train_idxs]\ntrain_labels = float.(onehotbatch(&#91;X&#91;i].ground_truth.class for i in train_idxs],1:10))\ntrain_dataset = make_minibatch(train_imgs,train_labels,batch_size)\n\nvalid_idxs = 49001:50000\nvalX = cat(&#91;getarray(X&#91;i].img) for i in valid_idxs]..., dims = 4) |> gpu\nvalY = float.(onehotbatch(&#91;X&#91;i].ground_truth.class for i in valid_idxs],1:10)) |> gpu\n\n# Defining the loss and accuracy functions\n\n@info \"VGG16 models instantiation ...\"\nm = vgg16()\n\nloss(x, y) = crossentropy(m(x) .+ \u03f5, y .+ \u03f5)\n\naccuracy(x, y) = mean(onecold(m(x)|>cpu, 1:10) .== onecold(y|>cpu, 1:10))\n\n# Defining the callback and the optimizer\n\nevalcb = throttle(() -> @info(accuracy(valX, valY)), 10)\n\nopt = ADAM()\n\n@info \"Training model...\"\n\n# used for plots\naccs = Array{Float32}(undef,0)\n@time begin\ndataset_len = length(train_dataset)\nfor i in 1:epochs\n  for (idx,dataset) in enumerate(train_dataset)\n    Flux.train!(loss,params(m),&#91;dataset],opt)\n    #Flux.train!(loss,params(m),&#91;dataset],opt,cb = evalcb)    \n    #if idx == dataset_len\n    acc = accuracy(valX,valY)\n    @info \"Epoch# $(i)\/$(epochs) - #$(idx)\/$(dataset_len) loss: $(loss(dataset...)), accuracy: $(acc)\"\n    push!(accs,acc)\n    #end\n    flush(log)\n  end\n  @save model_file m\nend\nend # end of @time\n# Fetch the test data from Metalhead and get it into proper shape.\n# CIFAR-10 does not specify a validation set so valimgs fetch the testdata instead of testimgs\ntX = valimgs(CIFAR10)\ntest_idxs = 1:10000\ntest_imgs = &#91;getarray(tX&#91;i].img) for i in test_idxs]\ntest_labels = float.(onehotbatch(&#91;tX&#91;i].ground_truth.class for i in test_idxs], 1:10))\ntest_dataset = make_minibatch(test_imgs,test_labels,batch_size)\n\ntest_accs = Array{Float32}(undef,0)\ndataset_len = length(test_dataset)\nfor (idx,dataset) in enumerate(test_dataset)\n  acc = accuracy(dataset...)\n  push!(test_accs,acc)\nend\n@info \"Test accuracy : $(mean(test_accs))\"\n@info \"End - $(now())\"\nclose(log)\nend\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>model-zoo : https:\/\/github.com\/FluxML\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10.jl \uc18c\uc2a4\ub97c \uc218\uc815\ud568 \uc218\uc815\ub41c \uc18c\uc2a4 : https:\/\/github.com\/mrchaos\/model-zoo\/blob\/master\/vision\/cifar10\/cifar10_gpu_minibatch.jl \uc774 \uae00\uc744 \ubcf4\uae30\uc804\uc5d0 \uba3c\uc800 \uc544\ub798 \uae00\uc744 \ucc38\uc870 \ud558\ub294 \uac83\uc774 \uc88b\ub2e4. \uadf8\ub9ac\uace0 cifar10.jl\uc744 \uc774\ud574\ud558\ub294\ub370 \ub3c4\uc6c0\uc774 \ub420\ub9cc\ud55c jupyter notebook(html) ==&gt; \uc5ec\uae30 cifar10_gpu_minibatch.jl (\uba87\uac00\uc9c0 \uc124\uba85\uc774 \ub354 \ucd94\uac00 \ub418\uc5b4 \uc788\uc74c) \uc704\uc758 \ucca8\ubd80\ud30c\uc77c\uc5d0 \uc18c\uc2a4\uc5d0\uc11c &#8220;CUDAnative.device!(3)&#8221; \uc744 \uc8fc\uc11d \ucc98\ub9ac \ud558\uba74 \uccab\ubc88\uc9f8 GPU\uc5d0\uc11c \uc2e4\ud589\ub41c\ub2e4<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[18,21],"tags":[],"_links":{"self":[{"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/posts\/2502"}],"collection":[{"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/julialang.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2502"}],"version-history":[{"count":3,"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/posts\/2502\/revisions"}],"predecessor-version":[{"id":2507,"href":"https:\/\/julialang.kr\/index.php?rest_route=\/wp\/v2\/posts\/2502\/revisions\/2507"}],"wp:attachment":[{"href":"https:\/\/julialang.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/julialang.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/julialang.kr\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}