export TZ=Asia/Tokyo # to set timezone for single terminal [Click]Tuesday, March 14, 2023
Saturday, October 1, 2022
Monitoring a Process Mem and CPU in Linux
Explained about Virtual Memory and Hows it works with Block diagram
https://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/9_VirtualMemory.html
https://www.digitalocean.com/community/tutorials/atop-command-in-linux
Useful option:
atop
- SYSCPU - The amount of CPU consumed by the process while system handling.
- USRCPU - The amount of CPU consumed by the process, during its running in user mode.
- VGROW - The amount of virtual memory that the process has occupied since the last value update.
- RGROW - The amount of resident (physical) memory grown since last value update.
- RDDSK - The size of data transferred during disk reads.
- WRDSK - The size of data transferred during disk writes.
- RUID - The real user ID under which the process is being executed.
- EUID - The effective user ID under which the process is being executed.
- ST - The current status of the process.
- EXC - The exit code after the process terminates
- THR - The number of threads within the process.
- S - The current status of the primary thread of the process.
- CPU - The percentage of CPU utilization of the entire process.
- CMD - The name of the process.
atop -m
- MINFLT - The number of minor page faults that have been solved by accessing data from free pages.
- MAJFLT - The number of major page faults that have been solved by especially retrieving data from disk.
- VSTEXT - The virtual memory occupied by the process text.
- VSLIBS - The virtual memory occupied by the shared libraries of the process.
- VDATA - The virtual memory size of the private data of the process.
- VSTACK - The virtual memory size of the private stack of the process.
- VSIZE - The total virtual memory size of the process.
- RSIZE - The total resident memory occupied by the process.
- MEM - The percentage of RAM consumed by the process.
Wednesday, July 6, 2022
Sunday, May 15, 2022
MongoDB Config, monitor, performance tuning
MongoDB Memory Usage, Management, & Requirements
MongoDB is not an in-memory database. Although it can be configured to run that way. But it makes liberal use of cache, meaning data records kept memory for fast retrieval, as opposed to on disk.
MongoDB, in its default configuration, will use will use the larger of either 256 MB or ½ of (ram – 1 GB) for its cache size.
You can limit the MongoDB cache size by adding the cacheSizeGB argument to the /etc/mongod.conf configuration file, as shown below. But MongoDB uses both its internal cache and the system’s file system cache. So limiting it in one place will cut make it consume more in the other. So that’s not a solution.
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
wiredTiger:
engineConfig:
cacheSizeGB: 1
Show Memory MongoDB Usage:
db.enableFreeMonitoring()
These metrics include cpu usage, etc. Above I show just the memory section. That is:
- resident—amount of actual physical memory (RAM) used by a process.
- virtual—RAM plus memory that has extended to the file system cache, i.e. virtual memory.
- mapped—MongoDB since version 3.2 does not do memory mapping of files anymore. That was used by the previous memory management module called MMAPv1. Now it uses WiredTiger by default.
To check your file system cache run free -k to show available virtual memory in kilobytes. Below is what my server looked like over 5 minute intervals when I ran the machine out of memory. The column to note is available. Divide the number shown by 1024*1024 to convert this number to gigabytes.
$free -k
total used free shared buff/cache available
Mem: 8173744 1088440 6753392 9052 331912 6816176
Swap: 0 0 0
total used free shared buff/cache available
Mem: 8173744 4654600 2663432 9080 855712 3249520
Swap: 0 0 0
total used free shared buff/cache available
Mem: 8173744 7992568 119508 9052 61668 45682Of course you can use top -p (mongod pid) to monitor memory too. VIRT and RES are virtual and resident memory. Below you see that mongod is using 91% of the system’s memory in this example. At that point the system became unresponsive.
You can also get memory usage statistics from MongoDB in the shell:
Copy
db.serverStatus().mem { "bits" : 64, "resident" : 907, "virtual" : 1897, "supported" : true, "mapped" : 0, "mappedWithJournal" : 0 }
Performance Analyse:
If you have even done C or C++ programming you will be familiar with malloc. That is the system function you call when you need to reserve memory for your machine. When MongoDB can no longer do that these warnings start showing up in /var/log/mongodb/mongod.log:
Copy
2019-02-19T03:44:18.738+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 202, after asserts: 686, after backgroundFlushing: 890, after connections: 1399, after dur: 1717, after extra_info: 2039, after freeMonitoring: 3003, after globalLock: 3389, after locks: 4073, after logicalSessionRecordCache: 4782, after network: 5520, after opLatencies: 6068, after opcounters: 6477, after opcountersRepl: 6818, after repl: 7441, after security: 7799, after storageEngine: 8616, after tcmalloc: 9676, after transactions: 10081, after transportSecurity: 10412, after wiredTiger: 22733, at end: 24170 }
Until it finally says:
Copy
2019-02-19T03:45:17.456+0000 F - [free_mon] out of memory.
You can also log into mongo and repeat the two commands below (you need to execute both to update the output) to watch your memory usage spike as your server goes downhill.
Copy
var mem = db.serverStatus().tcmalloc; mem.tcmalloc.formattedString
Here is the output. As you can see it has exhausted all the available memory on this 8GB machine.You can also log into mongo and repeat the two commands below (you need to execute both to update the output) to watch your memory usage spike as your server goes downhill.
Copy
------------------------------------------------ MALLOC: 73048776 ( 69.7 MiB) Bytes in use by application MALLOC: + 3792896 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 508776 ( 0.5 MiB) Bytes in central cache freelist MALLOC: + 978848 ( 0.9 MiB) Bytes in transfer cache freelist MALLOC: + 1399344 ( 1.3 MiB) Bytes in thread cache freelists MALLOC: + 1335552 ( 1.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 81064192 ( 77.3 MiB) Actual memory used (physical + swap) MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 81064192 ( 77.3 MiB) Virtual address space used MALLOC: MALLOC: 816 Spans in use MALLOC: 21 Thread heaps in use MALLOC: 4096 Tcmalloc page size ------------------------------------------------
You can also log into mongo and repeat the two commands below (you need to execute both to update the output) to watch your memory usage spike as your server goes downhill.
Useful Commands:
db.getCollectionNames().map(name => ({totalIndexSize: db.getCollection(name).stats().totalIndexSize, name: name})).sort((a, b) => a.totalIndexSize - b.totalIndexSize).forEach(printjson)
db.serverStatus().mem # show resident memory and virtual db.vehicledetails.latencyStats() # show latency for read and write ops
db.serverStatus() #in this u can see warning and asserts
db.vehicledetails.dataSize()
db.vehicledetails.storageSize() db.vehicledetails.totalSize() db.vehicledetails.totalIndexSize()
This show the oplog replication status :
rs.printReplicationInfo()configured oplog size: 2431.771484375MB
log length start to end: 173275secs (48.13hrs)
oplog first event time: Sun Oct 02 2022 00:30:10 GMT+0000 (GMT)
oplog last event time: Tue Oct 04 2022 00:38:05 GMT+0000 (GMT)
now: Tue Oct 04 2022 00:38:09 GMT+0000 (GMT)
https://www.alexbevi.com/blog/2020/01/26/what-is-mongodb-ftdc-aka-diagnostic-dot-data/
https://www.mongodb.com/docs/manual/reference/configuration-file-settings-command-line-options-mapping/#std-label-conf-file-command-line-mapping
Subscribe to:
Posts (Atom)