Mercurial > ec-dotfiles
annotate moefetch.sh @ 227:8b1f6f6b6a3b
Bugfixes:
- Login and password are not always used now. Added new global variable to check it.
- 'export PATH' is now only done if additional path(s) specified.
Not really enhancements:
- New coding style:
- Local variables now start with double underscores and global variables start with single underscore. Both lower letter.
- Function name is now snake_case. Camel_And_Snake_Case doesn't actually make sense :>
- More comments. Still need to add even more comments.
| author | edhoprima@gmail.com <edhoprima@gmail.com> |
|---|---|
| date | Mon, 14 Dec 2009 09:57:35 +0000 |
| parents | f8be4a3d3b4a |
| children | 5d3a0645b504 |
| rev | line source |
|---|---|
|
221
e891b563b797
wrong rule caused mass headache
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
220
diff
changeset
|
1 #!/bin/sh |
| 148 | 2 |
| 3 # Copyright (c) 2009, edogawaconan <me@myconan.net> | |
| 4 # | |
| 5 # Permission to use, copy, modify, and/or distribute this software for any | |
| 6 # purpose with or without fee is hereby granted, provided that the above | |
| 7 # copyright notice and this permission notice appear in all copies. | |
| 8 # | |
| 9 # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES | |
| 10 # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF | |
| 11 # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR | |
| 12 # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES | |
| 13 # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN | |
| 14 # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF | |
| 15 # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. | |
| 159 | 16 # |
| 17 # Lots of bugs here. Use with care | |
| 148 | 18 # USE WITH CARE |
| 159 | 19 # |
| 20 # what it does: fetch every picture that has the specified TAGS. | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
21 # requirement: wget, libxslt, openssl |
| 148 | 22 |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
23 # program additional paths for: cut, sed, wc, openssl, wget, xsltproc, grep |
| 159 | 24 ADDITIONAL_PATH= |
| 148 | 25 |
| 159 | 26 # default server address. Danbooru only! I do not take responsibility of stupidity. |
| 27 DEFAULT_SITE="moe.imouto.org" | |
| 148 | 28 |
| 29 # base directory. make sure it's writeable. I do not take responsibility if you don't own the folder and files as no check is done for this one. | |
| 159 | 30 # Structure is ${BASE_DIR}/<TAGS> |
| 31 # Absolute path only. | |
| 32 # Leave empty to use whatever folder you're running this at | |
| 193 | 33 BASE_DIR= |
| 148 | 34 |
| 35 # not user modifiable from here | |
| 36 | |
| 193 | 37 ### TODO: |
| 211 | 38 ### - sanity validator(?) |
| 193 | 39 ### - unified repository to save bandwidth |
| 40 ### - bug stomping | |
| 41 ### - sanity checking | |
| 227 | 42 ### - MOAR comments |
| 193 | 43 ### WILL BE FOR 0.3 |
| 44 | |
| 159 | 45 # useless welcome message. Also version |
| 46 Msg_Welcome() { | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
47 MOEFETCHVERSION="0.3-beta2" |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
48 echo "moefetch ${MOEFETCHVERSION} |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
49 Copyright (c) 2009 edogawaconan <me@myconan.net> |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
50 " |
| 159 | 51 } |
| 52 | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
53 get_md5() { cat -- "$1" | openssl dgst -md5; } |
|
226
f8be4a3d3b4a
- Fix basename to escape with / instead of -- (workaround for conflict between BSD/GNU/Solaris implementations)
edhoprima
parents:
225
diff
changeset
|
54 get_basename() { basename /"$1"; } |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
55 get_filename() { get_basename "${1%.*}"; } |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
56 get_cleantags() { printf "%s " "$@" | sed -e 's/\&/%26/g;s/=/%3D/g'; } |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
57 Is_NotMD5() { get_filename "$1" | sed -e 's/\([0-9a-f]\{32\}\)//g'; } |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
58 |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
59 |
| 159 | 60 # fatal error handler |
| 61 Err_Fatal() { | |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
62 echo " |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
63 Fatal error: ${1}" |
| 159 | 64 exit 1 |
| 65 } | |
| 66 | |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
67 Err_Impossible() { |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
68 echo " |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
69 Impossible error. Or you modified content of the working directories when the script is running. |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
70 Please report to moefetch.googlecode.com if you see this message (complete with entire run log)" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
71 exit 1 |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
72 } |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
73 |
| 159 | 74 # help message |
| 75 Err_Help() { | |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
76 echo "moefetch.sh COMMAND [-s SITE_URL] TAGS |
|
174
0948e76a57a1
added help. Bump to 0.1-beta2
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
173
diff
changeset
|
77 |
|
176
3d2ae9417273
even more improvement
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
175
diff
changeset
|
78 COMMAND: |
|
3d2ae9417273
even more improvement
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
175
diff
changeset
|
79 (quick)fetch: do a complete update. Add prefix quick to skip file checking |
|
3d2ae9417273
even more improvement
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
175
diff
changeset
|
80 check: get list of new files, clean up local folder and print total new files |
|
175
5b7a154dbd21
cosmetics fix for help message
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
174
diff
changeset
|
81 |
| 177 | 82 -s SITE_URL: Specify URL of the Danbooru powered site you want to leech from. Default is ${DEFAULT_SITE} |
|
175
5b7a154dbd21
cosmetics fix for help message
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
174
diff
changeset
|
83 |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
84 TAGS: Tags you want to download. Separated by spaces. Tag name follows standard Danbooru tagging scheme" |
| 193 | 85 exit 2 |
| 159 | 86 } |
| 87 | |
| 88 # generate link by transforming xml | |
| 89 Generate_Link() { | |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
90 echo " |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
91 Fetching XML file" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
92 tempnum=1000 |
| 224 | 93 iternum=1 |
|
195
652d9e268cee
test migration to printf
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
194
diff
changeset
|
94 > "${TEMP_PREFIX}-list" |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
95 while [ "${tempnum}" -ge 1000 ]; do |
| 227 | 96 __url="http://${SITE}/post/index.xml?tags=$(get_cleantags "${TAGS}")&offset=0&limit=1000&page=${iternum}" |
| 97 [ ${_use_login} -eq 1 ] && __url="${__url}&login=${LOGIN_USER}&password_hash=${LOGIN_PASS}" | |
| 98 wget "${__url}" -O "${TEMP_PREFIX}-xml" -e continue=off || Err_Fatal "Failed download catalog file" | |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
99 printf "Processing XML file... " |
|
213
dd95cf01602c
working around limit
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
212
diff
changeset
|
100 # xslt evilry |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
101 xsltproc - "${TEMP_PREFIX}-xml" <<EOF | sed 's/.*\(http.*\)\(\/[a-f0-9]\{32\}\).*\.\([^\.]*\)/\1\2.\3/g' | grep ^http > "${TEMP_PREFIX}-templist" |
| 148 | 102 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> |
| 103 <xsl:output method="xml" indent="yes"/> | |
| 104 <xsl:template match="post"> | |
| 105 <xsl:value-of select="@file_url" /> | |
| 106 </xsl:template> | |
| 107 </xsl:stylesheet> | |
| 108 EOF | |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
109 tempnum=$(echo $(wc -l < "${TEMP_PREFIX}-templist")) |
| 224 | 110 iternum=$((iternum + 1)) |
|
213
dd95cf01602c
working around limit
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
212
diff
changeset
|
111 cat "${TEMP_PREFIX}-templist" >> "${TEMP_PREFIX}-list" |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
112 echo "${tempnum} file(s) available" |
|
213
dd95cf01602c
working around limit
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
212
diff
changeset
|
113 done |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
114 numfiles=$(echo $(wc -l < "${TEMP_PREFIX}-list")) |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
115 echo "${numfiles} file(s) available on server" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
116 [ "${numfiles}" -gt 0 ] || Err_Fatal "Error in processing list or no files can be found with specified tag(s) or site." |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
117 } |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
118 |
| 148 | 119 |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
120 Progress_Init() { |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
121 _last="-" |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
122 printf "${_last}" |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
123 } |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
124 |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
125 Progress_Anim() { |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
126 case "${_last}" in |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
127 /) _last="-";; |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
128 -) _last=\\;; |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
129 \\) _last=\|;; |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
130 \|) _last="/";; |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
131 esac |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
132 printf "\b${_last}" |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
133 } |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
134 |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
135 Progress_Done() { printf "\bdone\n"; } |
|
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
136 |
| 200 | 137 # getting rid of ls (as per suggestion) |
| 138 Count_Files() { | |
| 224 | 139 numfiles=0 |
| 140 for dircontent in "${*}/"* "${*}/".*; do | |
| 141 if [ "${dircontent}" != "${*}/*" ] || [ -e "${dircontent}" ]; then | |
| 142 numfiles=$((numfiles + 1)) | |
| 200 | 143 fi |
| 144 done | |
| 224 | 145 echo $((numfiles - 2)) |
| 200 | 146 } |
| 147 | |
| 159 | 148 # check tools availability |
| 149 Check_Tools() { | |
| 150 # verify all programs required do indeed exist | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
151 commands="cut sed wc wget xsltproc xargs rm mkdir chown comm grep date openssl" |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
152 for cmd in ${commands} |
| 159 | 153 do |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
154 [ "$(command -v "${cmd}")" ] || Err_Fatal "${cmd} doesn't exist in ${PATH}" |
| 159 | 155 done |
| 156 } | |
| 157 | |
| 158 # verify required folders exist and writeable | |
| 159 Check_Folders(){ | |
| 223 | 160 [ -O "${BASE_DIR}" ] || Err_Fatal "You don't own ${BASE_DIR}. Please fix ${BASE_DIR} or run this script in your own directory." |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
161 for directory in temp trash deleted "${SITE_DIR}/${TARGET_DIR}"; do |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
162 if [ ! -d "${BASE_DIR}/${directory}" ]; then |
|
216
a869987c4646
did I say 'mess up'?
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
215
diff
changeset
|
163 mkdir -p "${BASE_DIR}/${directory}" || Err_Impossible |
| 159 | 164 fi |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
165 if [ ! -O "${BASE_DIR}/${directory}" ]; then |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
166 echo "You don't own the ${BASE_DIR}/${directory}, applying globally writeable permission on it" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
167 chmod -R u=rwX,g=rwX,o=rwX "${BASE_DIR}/${directory}" || Err_Impossible |
| 159 | 168 fi |
| 169 done | |
| 223 | 170 [ "$(Count_Files "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}")" -eq 0 ] && ISNEW=1 |
|
201
30d2fb656029
scrapping grep -vf
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
200
diff
changeset
|
171 for i in error ok list newlist templist; do |
| 196 | 172 touch "${TEMP_PREFIX}-${i}" || Fatal_Err "Error creating ${TEMP_PREFIX}-${i}. This shouldn't happen" |
| 159 | 173 done |
| 174 # | |
| 175 } | |
| 176 | |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
177 # Do some cleanup |
|
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
178 Cleanup_Repository() { |
|
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
179 # current dir: ${BASE_DIR}/${SITE_DIR}/${TARGET_DIR} |
|
207
17d816a63b4c
final progress version
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
206
diff
changeset
|
180 printf "Cleaning up repository folder... " |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
181 Progress_Init |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
182 trash_dir="${BASE_DIR}/trash/${trash_dir}/$(date -u "+${SITE_DIR}-${TARGET_DIR}-%Y%m%d-%H.%M")" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
183 trashes="These files have been moved to ${trash_dir}:" |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
184 has_trash= |
| 223 | 185 if [ ! -d "${trash_dir}" ]; then |
|
216
a869987c4646
did I say 'mess up'?
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
215
diff
changeset
|
186 mkdir -p "${trash_dir}" || Err_Impossible |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
187 else |
| 223 | 188 if [ ! -O "${trash_dir}" ]; then |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
189 chmod -R u=rwX,g=rwX,o=rwX "${BASE_DIR}/${directory}" || Err_Impossible |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
190 fi |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
191 fi |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
192 for trash in "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}/"* |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
193 do |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
194 is_trash= |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
195 if [ -d "${trash}" ] || [ -n "$(Is_NotMD5 "${trash}")" ] || [ -z "$(grep "$(get_basename "${trash}")" "${TEMP_PREFIX}-list")" ]; then |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
196 is_trash=1 |
|
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
197 has_trash=1 |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
198 mv -f -- "${trash}" "${trash_dir}" || Err_Impossible |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
199 trashes="${trashes} |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
200 $(get_basename "${trash}")" |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
201 fi |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
202 Progress_Anim |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
203 done |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
204 rmdir "${trash_dir}" 2>/dev/null |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
205 Progress_Done |
| 223 | 206 [ -n "${has_trash}" ] && echo "${trashes}" |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
207 } |
|
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
208 |
| 159 | 209 # check files correctness |
| 210 Check_Files() { | |
| 223 | 211 if [ ! -n "${ISNEW}" ]; then |
| 212 [ -z "${NOCLEAN}" ] && Cleanup_Repository | |
|
207
17d816a63b4c
final progress version
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
206
diff
changeset
|
213 printf "Checking for errors... " |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
214 Progress_Init |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
215 files_error="These files do not match its md5:" |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
216 files_notdanbooru="These files are not checked:" |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
217 has_err_filename= |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
218 has_err_md5= |
| 196 | 219 > "${TEMP_PREFIX}-error" |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
220 > "${TEMP_PREFIX}-ok" |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
221 for file in "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}/"* |
| 159 | 222 do |
| 223 | 223 if [ "${file}" != "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}/*" ]; then |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
224 if [ -n "$(Is_NotMD5 "${file}")" ] || [ -d "${file}" ]; then |
| 217 | 225 files_notdanbooru="${files_notdanbooru} |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
226 $(get_basename "${file}")" |
| 217 | 227 has_err_filename=1 |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
228 else |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
229 if [ "$(get_md5 "${file}")" = "$(get_filename "${file}")" ]; then |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
230 echo "$(get_basename "${file}")" >> "${TEMP_PREFIX}-ok" |
| 217 | 231 else |
| 232 rm "${file}" || Err_Fatal "Error removing ${file}" | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
233 echo "$(get_basename "${file}")" >> "${TEMP_PREFIX}-error" |
| 217 | 234 files_error="${files_error} |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
235 $(get_basename "${file}")" |
| 217 | 236 has_err_md5=1 |
| 237 fi | |
|
187
efd957294c8c
refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
186
diff
changeset
|
238 fi |
| 148 | 239 fi |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
240 Progress_Anim |
| 148 | 241 done |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
242 Progress_Done |
| 223 | 243 if [ ! -n "${has_err_md5}" ] && [ ! -n "${has_err_filename}" ]; then |
| 203 | 244 echo "All files OK" |
| 170 | 245 else |
| 223 | 246 if [ -n "${has_err_md5}" ]; then |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
247 echo "${files_error}" |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
248 echo "$(echo $(wc -l < "${TEMP_PREFIX}-error")) file(s) removed" |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
249 fi |
| 223 | 250 [ -n "${has_err_filename}" ] && echo "${files_notdanbooru}" |
| 170 | 251 fi |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
252 echo "$(echo $(wc -l < "${TEMP_PREFIX}-ok")) file(s) available locally" |
| 148 | 253 |
|
207
17d816a63b4c
final progress version
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
206
diff
changeset
|
254 printf "Generating list of new files... " |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
255 Progress_Init |
| 218 | 256 cp -f "${TEMP_PREFIX}-list" "${TEMP_PREFIX}-templist" |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
257 while read -r is_ok; do |
|
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
258 grep -v "${is_ok}" "${TEMP_PREFIX}-templist" > "${TEMP_PREFIX}-newlist" |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
259 cp -f "${TEMP_PREFIX}-newlist" "${TEMP_PREFIX}-templist" || Err_Impossible |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
260 Progress_Anim |
| 203 | 261 done < "${TEMP_PREFIX}-ok" |
|
205
2e866999c042
now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
204
diff
changeset
|
262 Progress_Done |
|
214
a6624fb9b317
major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
213
diff
changeset
|
263 echo "$(echo $(wc -l < "${TEMP_PREFIX}-newlist")) file(s) to be downloaded" |
| 148 | 264 else |
| 223 | 265 if [ -n "${ISQUICK}" ]; then |
|
207
17d816a63b4c
final progress version
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
206
diff
changeset
|
266 echo "Quick mode selected. Skipping check" |
| 152 | 267 else |
| 268 echo "Empty local repository" | |
| 269 fi | |
| 200 | 270 cat "${TEMP_PREFIX}-list" > "${TEMP_PREFIX}-newlist" |
| 148 | 271 fi |
| 272 } | |
| 273 | |
| 159 | 274 # start downloading the images |
| 275 Fetch_Images() { | |
| 223 | 276 if [ "$(echo $(wc -l < "${TEMP_PREFIX}-newlist"))" -eq 0 ]; then |
| 148 | 277 echo "No new file" |
| 278 else | |
|
207
17d816a63b4c
final progress version
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
206
diff
changeset
|
279 printf "Starting wget... " |
|
160
68227a30d0b3
forgot to fix Fetch_Images to reflect new folder naming scheme
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
159
diff
changeset
|
280 cd "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}" |
| 196 | 281 wget -e continue=on -bi "${TEMP_PREFIX}-newlist" -o "${TEMP_PREFIX}.log" |
| 148 | 282 fi |
| 283 } | |
| 284 | |
| 159 | 285 # initialize base variables and initial command check |
| 227 | 286 init() |
| 287 { | |
| 159 | 288 # path initialization |
| 227 | 289 # check if additional path is specified |
| 290 if [ -n "${ADDITIONAL_PATH}" ] | |
| 291 then | |
| 292 # insert the additional path | |
| 293 PATH="${ADDITIONAL_PATH}:${PATH}" | |
| 294 export PATH | |
| 295 fi | |
|
158
cba73f6a96bb
grep check. OpenSolaris' default grep doesn't support -f
edhoprima
parents:
157
diff
changeset
|
296 |
| 159 | 297 # misc variables |
| 166 | 298 ISQUICK= |
| 299 ISNEW= | |
| 227 | 300 # variable to check whether a login is used or not |
| 301 _use_login=0 | |
|
215
710082ce6788
major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
214
diff
changeset
|
302 |
| 227 | 303 # minimum number of arguments: 2 (command and tag). If less than two, exit and print help message |
| 159 | 304 [ $# -lt 2 ] && Err_Help |
| 305 case "$1" in | |
|
174
0948e76a57a1
added help. Bump to 0.1-beta2
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
173
diff
changeset
|
306 check|fetch|quickfetch) |
| 159 | 307 echo "Starting..." |
| 308 JOB="$1" | |
| 309 ;; | |
| 310 *) | |
| 311 Err_Help | |
| 312 ;; | |
| 313 esac | |
| 314 shift | |
| 315 SITE= | |
|
185
6d926d4b3c5a
initial clean system support
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
184
diff
changeset
|
316 TAGS= |
| 227 | 317 __has_pass=0 |
| 318 __has_user=0 | |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
319 while getopts "s:(site)n(noclean)u:(user)p:(password)" opt |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
320 do |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
321 case "$opt" in |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
322 s) SITE="$OPTARG";; |
|
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
323 n) NOCLEAN=1;; |
| 227 | 324 p) |
| 325 LOGIN_PASS=$(printf "%s" "$OPTARG" | openssl dgst -sha1) | |
| 326 __has_pass=1 | |
| 327 ;; | |
| 328 u) | |
| 329 LOGIN_USER="$OPTARG" | |
| 330 __has_user=1 | |
| 331 ;; | |
|
185
6d926d4b3c5a
initial clean system support
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
184
diff
changeset
|
332 esac |
|
6d926d4b3c5a
initial clean system support
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
184
diff
changeset
|
333 done |
|
225
265a9ca47a19
- Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents:
224
diff
changeset
|
334 TAGS="$@" |
| 223 | 335 [ -n "${SITE}" ] || SITE=${DEFAULT_SITE} |
| 336 [ -n "${TAGS}" ] || Err_Fatal "No tag specified" | |
|
181
d3b7927bdb2b
restructuring and add check if the xml is processed properly
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
180
diff
changeset
|
337 # Get base folder - default, current folder or fallback to ${HOME} |
| 223 | 338 [ -n "${BASE_DIR}" ] || BASE_DIR=${PWD} |
| 339 [ -n "${BASE_DIR}" ] || BASE_DIR=${HOME} | |
| 227 | 340 [ -n "$(echo "${BASE_DIR}" | cut -c1 | grep \/)" ] || BASE_DIR="/${BASE_DIR}" |
| 341 # see if both pass and use are set. If they're set, switch _use_login variable content to 1. | |
| 342 [ ${__has_pass} -eq 1 -a ${__has_user} -eq 1 ] && _use_login=1 | |
|
181
d3b7927bdb2b
restructuring and add check if the xml is processed properly
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
180
diff
changeset
|
343 |
| 159 | 344 echo "Tags: ${TAGS}" |
| 345 # slash is not wanted for folder name | |
| 193 | 346 TARGET_DIR=$(echo "${TAGS}" | sed -e 's/\//_/g') |
| 347 SITE_DIR=$(echo "${SITE}" | sed -e 's/\/$//g;s/\//_/g') | |
|
195
652d9e268cee
test migration to printf
edhoprima@gmail.com <edhoprima@gmail.com>
parents:
194
diff
changeset
|
348 TEMP_PREFIX="${BASE_DIR}/temp/${SITE_DIR}-${TARGET_DIR}" |
| 159 | 349 } |
| 148 | 350 |
| 227 | 351 main() |
| 352 { | |
| 353 # initialization | |
| 354 Msg_Welcome | |
| 355 init "$@" | |
| 356 Check_Tools | |
| 357 Check_Folders | |
|
158
cba73f6a96bb
grep check. OpenSolaris' default grep doesn't support -f
edhoprima
parents:
157
diff
changeset
|
358 |
| 148 | 359 |
| 227 | 360 # let's do the job! |
| 361 case "${JOB}" in | |
| 362 check) | |
| 363 Generate_Link | |
| 364 Check_Files | |
| 365 ;; | |
| 366 fetch) | |
| 367 Generate_Link | |
| 368 Check_Files | |
| 369 Fetch_Images | |
| 370 ;; | |
| 371 quickfetch) | |
| 372 ISNEW=1 | |
| 373 ISQUICK=1 | |
| 374 Generate_Link | |
| 375 Check_Files | |
| 376 Fetch_Images | |
| 377 ;; | |
| 378 esac | |
| 379 } | |
| 380 | |
| 381 # call the main routine! | |
| 382 main "$@" | |
| 383 |