.tif내 디렉토리에는 약 130000개의 파일이 있습니다. tesseract를 사용하여 텍스트를 추출해야 합니다. 메모리 드라이브를 만들고 tmpfs모든 파일을 그곳으로 옮겼습니다. 내 내부 페이지 중 하나는 다음과 같습니다.
printf "%03d\n" {1..163} | parallel find . -name 'k-{}\*' -print0 | parallel -j100 -0 'tesseract {/} /mnt/ramdisk/output/{/.} > /dev/null 2>&1'
내 컴퓨터는 4개의 vCPU 코어와 16GB RAM을 갖춘 Ubuntu 16.04를 실행하는 클라우드 가상 머신입니다. 100개가 넘는 작업을 병렬로 실행하고 있지만 분당 25~30개 이상의 파일을 가져올 수 없습니다.
디스크 I/O가 병목 현상을 일으키나요? 나는 이 주제에 대해 거의 아무것도 모른다. 분당 더 많은 파일을 실행하려면 어떻게 해야 합니까?
ps -e출력 은 다음과 같습니다 .
21002 pts/0 00:00:06 tesseract <defunct>
21010 pts/0 00:00:05 tesseract <defunct>
21023 pts/0 00:00:06 tesseract <defunct>
21035 pts/0 00:00:04 tesseract <defunct>
21039 pts/0 00:00:06 tesseract <defunct>
21040 pts/0 00:00:04 tesseract <defunct>
21044 pts/0 00:00:00 tesseract <defunct>
21056 pts/0 00:00:04 tesseract <defunct>
21065 pts/0 00:00:05 tesseract <defunct>
21067 pts/0 00:00:06 tesseract <defunct>
21068 pts/0 00:00:06 tesseract <defunct>
21080 pts/0 00:00:06 tesseract <defunct>
21081 pts/0 00:00:06 tesseract <defunct>
21082 pts/0 00:00:05 tesseract <defunct>
21083 pts/0 00:00:06 tesseract <defunct>
21084 pts/0 00:00:06 tesseract <defunct>
21085 pts/0 00:00:00 tesseract <defunct>
21087 pts/0 00:00:06 tesseract <defunct>
21090 pts/0 00:00:05 tesseract <defunct>
21094 pts/0 00:00:00 tesseract <defunct>
21099 pts/0 00:00:05 tesseract <defunct>
21100 pts/0 00:00:02 tesseract <defunct>
21102 pts/0 00:00:06 tesseract <defunct>
21103 pts/0 00:00:05 tesseract <defunct>
21104 pts/0 00:00:06 tesseract <defunct>
21106 pts/0 00:00:08 tesseract <defunct>
21107 pts/0 00:00:06 tesseract <defunct>
21108 pts/0 00:00:04 tesseract <defunct>
21110 pts/0 00:00:06 tesseract <defunct>
21111 pts/0 00:00:08 tesseract <defunct>
21112 pts/0 00:00:06 tesseract <defunct>
21113 pts/0 00:00:04 tesseract <defunct>
21114 pts/0 00:00:06 tesseract <defunct>
21115 pts/0 00:00:06 tesseract <defunct>
21117 pts/0 00:00:01 tesseract <defunct>
21118 pts/0 00:00:00 tesseract <defunct>
21119 pts/0 00:00:04 tesseract <defunct>
21128 pts/0 00:00:05 tesseract <defunct>
21129 pts/0 00:00:00 tesseract <defunct>
21134 pts/0 00:00:06 tesseract <defunct>
21135 pts/0 00:00:04 tesseract <defunct>
21137 pts/0 00:00:06 tesseract <defunct>
21138 pts/0 00:00:00 tesseract <defunct>
21141 pts/0 00:00:06 tesseract <defunct>
21144 pts/0 00:00:05 tesseract <defunct>
21145 pts/0 00:00:06 tesseract <defunct>
21150 pts/0 00:00:06 tesseract <defunct>
21151 pts/0 00:00:02 tesseract <defunct>
21152 pts/0 00:00:06 tesseract <defunct>
21154 pts/0 00:00:00 tesseract <defunct>
21155 pts/0 00:00:05 tesseract <defunct>
21158 pts/0 00:00:05 tesseract <defunct>
21159 pts/0 00:00:06 tesseract <defunct>
21160 pts/0 00:00:06 tesseract <defunct>
21165 pts/0 00:00:06 tesseract <defunct>
21167 pts/0 00:00:05 tesseract <defunct>
21173 pts/0 00:00:00 tesseract <defunct>
21175 pts/0 00:00:00 tesseract <defunct>
21181 pts/0 00:00:06 tesseract <defunct>
21184 pts/0 00:00:05 tesseract <defunct>
21185 pts/0 00:00:06 tesseract <defunct>
21187 pts/0 00:00:05 tesseract <defunct>
21191 pts/0 00:00:06 tesseract <defunct>
21196 pts/0 00:00:00 tesseract <defunct>
21197 pts/0 00:00:05 tesseract <defunct>
21198 pts/0 00:00:06 tesseract <defunct>
21199 pts/0 00:00:08 tesseract
21200 pts/0 00:00:00 tesseract <defunct>
21201 pts/0 00:00:06 tesseract <defunct>
21202 pts/0 00:00:03 tesseract <defunct>
21204 pts/0 00:00:08 tesseract
21207 pts/0 00:00:06 tesseract <defunct>
21210 pts/0 00:00:00 tesseract <defunct>
21214 pts/0 00:00:00 tesseract <defunct>
21215 pts/0 00:00:06 tesseract <defunct>
21217 pts/0 00:00:05 tesseract <defunct>
21218 pts/0 00:00:03 tesseract <defunct>
21221 pts/0 00:00:06 tesseract <defunct>
21224 ? 00:00:00 kworker/0:1
21229 pts/0 00:00:06 tesseract <defunct>
21239 pts/0 00:00:06 tesseract <defunct>
21240 pts/0 00:00:04 tesseract <defunct>
21241 pts/0 00:00:06 tesseract <defunct>
21242 pts/0 00:00:02 tesseract <defunct>
21245 pts/0 00:00:06 tesseract <defunct>
21246 pts/0 00:00:05 tesseract <defunct>
21247 pts/0 00:00:04 tesseract <defunct>
21248 pts/0 00:00:00 tesseract <defunct>
21250 pts/0 00:00:06 tesseract <defunct>
21251 pts/0 00:00:05 tesseract <defunct>
21252 pts/0 00:00:01 tesseract <defunct>
21253 pts/0 00:00:06 tesseract <defunct>
21265 pts/0 00:00:06 tesseract <defunct>
21268 pts/0 00:00:06 tesseract <defunct>
21269 pts/0 00:00:06 tesseract <defunct>
21270 pts/0 00:00:01 tesseract <defunct>
21277 pts/0 00:00:06 tesseract <defunct>
21285 pts/0 00:00:05 tesseract
21289 pts/0 00:00:07 tesseract <defunct>
21291 pts/0 00:00:07 tesseract
21292 pts/0 00:00:06 tesseract
21293 pts/0 00:00:06 tesseract
21297 pts/0 00:00:06 tesseract
21299 pts/0 00:00:05 tesseract
21300 pts/0 00:00:06 tesseract <defunct>
21302 pts/0 00:00:05 tesseract
21303 pts/0 00:00:06 tesseract <defunct>
21308 pts/0 00:00:05 tesseract <defunct>
21320 pts/0 00:00:05 tesseract
21322 pts/0 00:00:00 tesseract <defunct>
21324 pts/0 00:00:05 tesseract <defunct>
21329 pts/0 00:00:05 tesseract <defunct>
21330 pts/0 00:00:05 tesseract <defunct>
21333 pts/0 00:00:05 tesseract
21334 pts/0 00:00:05 tesseract
21335 pts/0 00:00:06 tesseract
21337 pts/0 00:00:00 tesseract <defunct>
21338 pts/0 00:00:04 tesseract <defunct>
21340 pts/0 00:00:04 tesseract
21341 pts/0 00:00:05 tesseract
21344 pts/0 00:00:05 tesseract
21353 pts/0 00:00:06 tesseract <defunct>
21362 pts/0 00:00:06 tesseract <defunct>
21363 pts/0 00:00:00 tesseract <defunct>
21366 pts/0 00:00:03 tesseract
21372 pts/0 00:00:04 tesseract
21373 pts/0 00:00:04 tesseract
21377 pts/0 00:00:00 tesseract <defunct>
21378 pts/0 00:00:05 tesseract
21382 pts/0 00:00:05 tesseract <defunct>
21384 pts/0 00:00:04 tesseract
21385 pts/0 00:00:05 tesseract
21391 pts/0 00:00:05 tesseract <defunct>
21393 pts/0 00:00:01 tesseract <defunct>
21394 pts/0 00:00:04 tesseract
21396 pts/0 00:00:01 tesseract <defunct>
21397 pts/0 00:00:02 tesseract <defunct>
21399 pts/0 00:00:05 tesseract
21401 pts/0 00:00:06 tesseract <defunct>
21403 pts/0 00:00:00 tesseract <defunct>
21404 pts/0 00:00:01 tesseract <defunct>
21405 pts/0 00:00:00 tesseract <defunct>
21407 pts/0 00:00:04 tesseract
21408 pts/0 00:00:00 tesseract <defunct>
21409 pts/0 00:00:05 tesseract
21413 pts/0 00:00:02 tesseract <defunct>
21414 pts/0 00:00:04 tesseract
21415 pts/0 00:00:05 tesseract
21416 pts/0 00:00:04 tesseract
21417 pts/0 00:00:00 tesseract <defunct>
21418 pts/0 00:00:03 tesseract
21419 pts/0 00:00:03 tesseract
21422 pts/0 00:00:03 tesseract
21424 pts/0 00:00:04 tesseract
21426 pts/0 00:00:03 tesseract
21427 pts/0 00:00:04 tesseract
21429 pts/0 00:00:05 tesseract
21434 pts/0 00:00:01 tesseract <defunct>
21437 pts/0 00:00:04 tesseract
21438 pts/0 00:00:00 nest.sh
21440 pts/0 00:00:02 tesseract
21445 pts/0 00:00:03 tesseract
21446 pts/0 00:00:04 tesseract
21447 pts/0 00:00:04 tesseract
21448 pts/0 00:00:02 tesseract
21449 pts/0 00:00:03 tesseract
21450 pts/0 00:00:04 tesseract
21451 pts/0 00:00:04 tesseract
21452 pts/0 00:00:02 tesseract <defunct>
21454 pts/0 00:00:03 tesseract
21455 pts/0 00:00:03 tesseract
21457 pts/0 00:00:00 perl
21467 ? 00:00:00 kworker/2:2
21471 pts/0 00:00:03 tesseract
21472 pts/0 00:00:03 tesseract
21475 pts/0 00:00:00 tesseract <defunct>
21480 ? 00:00:00 kworker/u8:2
21485 pts/0 00:00:02 tesseract
21487 pts/0 00:00:03 tesseract
21488 pts/0 00:00:02 tesseract
21489 pts/0 00:00:03 tesseract
21493 pts/0 00:00:02 tesseract
21496 pts/0 00:00:03 tesseract
21498 pts/0 00:00:03 tesseract
21501 pts/0 00:00:03 tesseract
21508 pts/0 00:00:02 tesseract <defunct>
21510 pts/0 00:00:03 tesseract
21516 pts/0 00:00:00 tesseract <defunct>
21519 pts/0 00:00:02 tesseract
21521 pts/0 00:00:03 tesseract
21523 pts/0 00:00:02 tesseract
21531 pts/0 00:00:02 tesseract
21532 pts/0 00:00:02 tesseract
21536 pts/0 00:00:00 tesseract <defunct>
21537 pts/0 00:00:02 tesseract
21543 pts/0 00:00:01 tesseract
21545 pts/0 00:00:02 tesseract
21549 pts/0 00:00:02 tesseract
21553 pts/0 00:00:02 tesseract
21556 pts/0 00:00:00 tesseract <defunct>
21560 pts/0 00:00:00 tesseract
21561 pts/0 00:00:01 tesseract
21562 pts/0 00:00:01 tesseract
21566 pts/0 00:00:01 tesseract
21569 pts/0 00:00:01 tesseract
21570 pts/0 00:00:01 tesseract
21575 pts/0 00:00:00 tesseract <defunct>
21582 pts/0 00:00:02 tesseract
21583 pts/0 00:00:00 tesseract
21585 pts/0 00:00:01 tesseract
21586 pts/0 00:00:01 tesseract
21590 pts/0 00:00:01 tesseract
21595 pts/0 00:00:00 tesseract <defunct>
21596 pts/0 00:00:00 tesseract
21597 pts/0 00:00:01 tesseract
21598 pts/0 00:00:01 tesseract
21599 pts/0 00:00:00 tesseract
21600 pts/0 00:00:00 tesseract <defunct>
21606 pts/0 00:00:00 tesseract
21610 pts/0 00:00:00 tesseract
21611 pts/0 00:00:01 tesseract
21612 pts/0 00:00:00 tesseract
21615 pts/0 00:00:01 tesseract
21620 pts/0 00:00:01 tesseract
21621 pts/0 00:00:00 tesseract
21622 pts/0 00:00:00 tesseract
21624 pts/0 00:00:00 tesseract
21625 pts/0 00:00:00 tesseract
21626 pts/0 00:00:00 tesseract
21639 pts/0 00:00:01 tesseract
21641 pts/0 00:00:01 tesseract
21646 ? 00:00:00 mysql-systemd-s
21648 pts/0 00:00:00 tesseract
21653 pts/0 00:00:00 tesseract
21654 pts/0 00:00:00 tesseract
21655 pts/0 00:00:00 tesseract
21660 pts/0 00:00:00 tesseract
21665 pts/0 00:00:00 tesseract
21666 pts/0 00:00:00 tesseract
21670 pts/0 00:00:00 tesseract
21671 pts/0 00:00:00 tesseract
21674 pts/0 00:00:00 tesseract
21675 pts/0 00:00:00 tesseract
21677 pts/0 00:00:00 tesseract
21682 pts/0 00:00:00 tesseract
21683 pts/0 00:00:00 tesseract
21684 pts/0 00:00:00 tesseract
21688 pts/0 00:00:00 tesseract
21690 pts/0 00:00:00 tesseract
21691 pts/0 00:00:00 tesseract
21692 pts/0 00:00:00 tesseract
21693 pts/0 00:00:00 tesseract
21696 pts/0 00:00:00 tesseract
21697 pts/0 00:00:00 tesseract
21699 pts/0 00:00:00 tesseract
21703 pts/0 00:00:00 tesseract
21705 pts/0 00:00:00 tesseract
21708 pts/0 00:00:00 tesseract
21709 pts/0 00:00:00 tesseract
21710 pts/0 00:00:00 tesseract
21711 pts/0 00:00:00 tesseract
21712 pts/0 00:00:00 tesseract
21713 pts/0 00:00:00 tesseract
21714 pts/0 00:00:00 tesseract
21716 pts/0 00:00:00 tesseract
21717 pts/0 00:00:00 tesseract
21720 pts/0 00:00:00 tesseract
21724 pts/0 00:00:00 tesseract
21727 pts/0 00:00:00 tesseract
21730 pts/0 00:00:00 tesseract
21732 pts/0 00:00:00 tesseract
21733 pts/0 00:00:00 tesseract
21736 pts/0 00:00:00 tesseract
21739 pts/0 00:00:00 tesseract
21742 pts/0 00:00:00 tesseract
21745 pts/0 00:00:00 tesseract
21752 pts/0 00:00:00 tesseract
21755 pts/0 00:00:00 tesseract
21756 pts/0 00:00:00 tesseract
21758 pts/0 00:00:00 tesseract
21760 pts/0 00:00:00 tesseract
21762 pts/0 00:00:00 tesseract
21764 pts/0 00:00:00 tesseract
21765 pts/0 00:00:00 tesseract
21766 pts/0 00:00:00 tesseract
21769 pts/0 00:00:00 tesseract
21773 pts/0 00:00:00 tesseract
21778 pts/0 00:00:00 tesseract
21779 pts/0 00:00:00 tesseract
21780 pts/0 00:00:00 tesseract
21782 pts/0 00:00:00 tesseract
21784 pts/0 00:00:00 sleep
21785 pts/0 00:00:00 sleep
21786 pts/0 00:00:00 sleep
21787 pts/0 00:00:00 tesseract
21788 pts/0 00:00:00 bash
21789 pts/0 00:00:00 sleep
21790 pts/0 00:00:00 sleep
21791 pts/0 00:00:00 sleep
21792 pts/0 00:00:00 sleep
21794 pts/0 00:00:00 sleep
21795 pts/0 00:00:00 tesseract
21796 pts/0 00:00:00 tesseract
21801 pts/0 00:00:00 tesseract
21803 pts/0 00:00:00 bash
21804 pts/0 00:00:00 bash
21807 pts/0 00:00:00 sleep
21808 pts/0 00:00:00 tesseract
21809 pts/0 00:00:00 tesseract
21810 pts/0 00:00:00 bash
21811 pts/0 00:00:00 sleep
21812 pts/0 00:00:00 sleep
21813 pts/0 00:00:00 perl
21814 ? 00:00:00 sshd
21815 pts/0 00:00:00 perl
21816 ? 00:00:00 sleep
21817 pts/1 00:00:00 ps
답변1
이미 tmpfs를 사용하고 있다면 IO로 인해 병목 현상이 발생해서는 안 됩니다. 스크립트의 명백한 문제점은 동시에 너무 많은 작업을 실행하려고 한다는 것입니다. 작업이 tmpfs로 인해 아무 것도 기다리면 안 되므로 보유한 코어 수(아마도 +1)와 동일한 수의 병렬 작업을 사용해야 합니다. 또한 더 빠르게 테스트할 수 있도록 테스트 세트를 200개 파일로 줄이세요.
답변2
이 I/O의 원인을 파악해야 합니다. 읽기 및 쓰기 tmpfs로 인해 I/O가 발생 하지 않습니다 .
:> cd /run/user/1000
:> LC_ALL=C df -h .
Filesystem Size Used Avail Use% Mounted on
tmpfs 2.4G 128M 2.3G 6% /run/user/1000
:> dd if=/dev/urandom of=bigfile bs=1M count=100
:> ls -lh bigfile
-rw-r--r-- 1 hl hauke 100M 30. Dez 16:07 bigfile
:> while true; do cat <bigfile >bigfilecopy || break; echo -n .; done
1GiB/s 이상의 속도로 데이터를 읽고 썼지만 topI/O 대기 시간은 0.0%로 나타났습니다.
프로세스 시작은 초당 25번만 발생하고 어쨌든 페이지 캐시에 의해 재정의되어야 하므로 문제가 되지 않습니다.
I/O의 원인을 찾아 제거할 수 없는 경우에는 서로 다른 소수의 병렬 프로세스의 I/O 대기를 비교해야 합니다. 이미 말했듯이 일반적으로 CPU 코어보다 더 많은 프로세스를 갖는 것은 의미가 없습니다. 그러나 2개 또는 3개의 프로세스로 인해 I/O 대기 시간이 매우 짧고 4개의 프로세스로 인해 로드가 훨씬 높을 수 있으므로 3개의 병렬 프로세스를 사용하는 것이 더 빠를 수도 있습니다.