Hi Arix,

This is happening due to the excessive writes on the VM disk, you can check using these commands.

show vmhost snapshot
show vmhost resource-usage

Check your system storage in general, any logs that are rotating frequently are a good insight/red flag and last but not least check for any trace options and deactivate them as they are really storage consuming. 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

No traceoption log file(s) are on...

> show vmhost resource-usage
Compute cluster: rainier-re-cc

  Compute node: rainier-re-cn
   CPU Usage
   =========
   13:29:21     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
   13:29:21     all    5.09    0.00    3.80    0.02    0.00    0.03    0.00   65.95   25.10
   13:29:21       0   15.32    0.00    9.38    0.08    0.00    0.16    0.00    0.00   75.07
   13:29:21       1   15.22    0.00    9.17    0.06    0.00    0.00    0.00    0.00   75.55
   13:29:21       2    0.01    0.00    1.67    0.00    0.00    0.00    0.00   98.33    0.00
   13:29:21       3    0.00    0.00    0.80    0.00    0.00    0.00    0.00   99.20    0.00
   13:29:21       4    0.00    0.00    0.81    0.00    0.00    0.00    0.00   99.19    0.00
   13:29:21       5    0.00    0.00    0.99    0.00    0.00    0.00    0.00   99.00    0.00

   Memory Usage
   ============
                total       used       free     shared    buffers     cached
   Mem:         63977      50218      13284          0        202        272
   Swap:            0          0          0

   Disk Usage
   ==========
   Filesystem                    Size  Used Avail Use% Mounted on
   rootfs                        3.3G   57M  3.0G   2% /
   /dev/sda3                     2.0G  1.5G  318M  83% /.old
   tmpfs                          32G  1.2M   32G   1% /.old/run
   none                           32G  4.0K   32G   1% /.old/dev
   tmpfs                          32G     0   32G   0% /.old/tmp
   tmpfs                          32G     0   32G   0% /.old/tdev
   /dev/mapper/jvg_P-jlvmrootrw  3.3G   57M  3.0G   2% /.old/tdev/.union/.s
   unionfs                       3.3G   57M  3.0G   2% /
   none                           32G  4.0K   32G   1% /dev
   tmpfs                          32G  208K   32G   1% /run
   tmpfs                          32G   20K   32G   1% /var/volatile
   /dev/mapper/jvg_P-jlvmjunos    32G   18G   13G  55% /junos
   /dev/mapper/jvg_P-jlvmvm      6.4G  4.5G  1.6G  75% /vm
   /dev/mapper/jvg_P-jlvmspare   6.8M   45K  6.2M   2% /spare
   cgroup                         32G     0   32G   2% /sys/fs/cgroup
   unionfs                       3.3G   57M  3.0G   2% /run/named-chroot/etc/bind
   tmpfs                          32G  208K   32G   3% /run/named-chroot/var/run/named
   tmpfs                          32G  208K   32G   4% /run/named-chroot/var/run/bind
   unionfs                       3.3G   57M  3.0G   3% /run/named-chroot/var/cache/bind
   unionfs                       3.3G   57M  3.0G   6% /run/named-chroot/etc/localtime
   none                           32G  4.0K   32G   5% /run/named-chroot/dev/random
   none                           32G  4.0K   32G   8% /run/named-chroot/dev/zero
   none                           32G  4.0K   32G   1% /run/named-chroot/dev/null

Hey Arix,

Any recent configuration changes?
Do you have SNMP enabled?

Can you double-check the traceoptions and the system logging?

show configuration | display set | match traceoption
show configuration system syslog | display set
show system processes extensive | except 0.0

show log ?

If you are not able to find what is causing the issue from the surface you will have to go into the vmhost, look for huge files and manually delete them: https://kb.juniper.net/InfoCenter/index?page=content&id=KB32711&act=login

The alarm is activated when the writes rate is crossing the threshold and clears off after the write rate comes down. This can be seen in a scaled setup with too many logs enabled.

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Hello Arix, 

Could you please share the output of below commands:
show log dcd
show log dfwd
show log messages | match "DFWD_TRASHED_RED_ZONE"

Also was there any config change made in the due course of time?

Regards, 

Sharat Ainapur

>sh log dfwd
error: could not resolve file: dfwd
>show log dcd | last | no-more
Mar 25 10:12:21 2489 dcd.c:3340 run_daemon() RSVRD : Going idle, 3 sync writes, 0 sync reads, 4 ifstate msgs, 4 ifstate reads, 4 async ifd msgs, 0 async rtb msgs,  0 async bd msgs, 0 async mesh group msgs, 3524788 usec
Mar 25 10:12:26 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for IDLE                : user 0.000 s, sys 0.000 s, wall 5.005 s
Mar 25 10:12:26 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is CONFIG_RETRY
Mar 25 10:12:30 2489 rtsock_io.c:1991 rtsock_send() ERROR : rtsock_send Sucessive failures Can't assign requested address for ADD_AND_GET giving UP !
Mar 25 10:12:30 2489 dcd_sock.c:1689 dcd_process_config_write() ERROR : DCD_CONFIG_WRITE_FAILED: IFA em0.0 (family=2) configuration write failed for an IFA ADD: Can't assign requested address
Mar 25 10:12:30 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for CONFIG_RETRY        : user 0.000 s, sys 0.000 s, wall 4.005 s
Mar 25 10:12:30 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is IDLE
Mar 25 10:12:30 2489 dcd.c:3340 run_daemon() RSVRD : Going idle, 2 sync writes, 0 sync reads, 0 ifstate msgs, 0 ifstate reads, 0 async ifd msgs, 0 async rtb msgs,  0 async bd msgs, 0 async mesh group msgs, 4007794 usec
Mar 25 10:12:35 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for IDLE                : user 0.000 s, sys 0.000 s, wall 5.006 s
Mar 25 10:12:35 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is CONFIG_RETRY
Mar 25 10:12:39 2489 rtsock_io.c:1991 rtsock_send() ERROR : rtsock_send Sucessive failures Can't assign requested address for ADD_AND_GET giving UP !
Mar 25 10:12:39 2489 dcd_sock.c:1689 dcd_process_config_write() ERROR : DCD_CONFIG_WRITE_FAILED: IFA em0.0 (family=2) configuration write failed for an IFA ADD: Can't assign requested address
Mar 25 10:12:39 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for CONFIG_RETRY        : user 0.000 s, sys 0.000 s, wall 4.004 s
Mar 25 10:12:39 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is IDLE
Mar 25 10:12:39 2489 dcd.c:3340 run_daemon() RSVRD : Going idle, 2 sync writes, 0 sync reads, 0 ifstate msgs, 0 ifstate reads, 0 async ifd msgs, 0 async rtb msgs,  0 async bd msgs, 0 async mesh group msgs, 4004982 usec
Mar 25 10:12:44 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for IDLE                : user 0.000 s, sys 0.000 s, wall 5.005 s
Mar 25 10:12:44 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is CONFIG_RETRY
Mar 25 10:12:48 2489 rtsock_io.c:1991 rtsock_send() ERROR : rtsock_send Sucessive failures Can't assign requested address for ADD_AND_GET giving UP !
Mar 25 10:12:48 2489 dcd_sock.c:1689 dcd_process_config_write() ERROR : DCD_CONFIG_WRITE_FAILED: IFA em0.0 (family=2) configuration write failed for an IFA ADD: Can't assign requested address
Mar 25 10:12:48 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for CONFIG_RETRY        : user 0.000 s, sys 0.001 s, wall 4.005 s
Mar 25 10:12:48 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is IDLE
Mar 25 10:12:48 2489 dcd.c:3340 run_daemon() RSVRD : Going idle, 2 sync writes, 0 sync reads, 4 ifstate msgs, 4 ifstate reads, 4 async ifd msgs, 0 async rtb msgs,  0 async bd msgs, 0 async mesh group msgs, 4009599 usec
Mar 25 10:12:48 2489 dcd.c:3517 run_daemon() RSVRD : Received SIGVTALRM, starting mini-parse.
Mar 25 10:12:48 2489 dcd.c:3222 run_daemon() RSVRD : Pending overlay config request now being serviced.
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.vrrp.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.autod.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.pppoed.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.dood.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.isdnd.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.dhcpd.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.jdhcpd.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.jdhcpdv6.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.jdhcpd_pd.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.ufd.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.vnode_lag.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.vnode.conf'
Mar 25 10:12:48 2489 lexer.c:425 lexer_begin_file() RSVRD : couldn't open include file '/var/etc/dcd.vnode.ctrl_ifl.conf'
Mar 25 10:12:48 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for IDLE                : user 0.002 s, sys 0.008 s, wall 0.070 s
Mar 25 10:12:48 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is CONFIG_MINI
Mar 25 10:12:48 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for CONFIG_MINI         : user 0.012 s, sys 0.000 s, wall 0.014 s
Mar 25 10:12:48 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is IDLE
Mar 25 10:12:48 2489 dcd.c:3340 run_daemon() RSVRD : Going idle, 1 sync writes, 0 sync reads, 0 ifstate msgs, 0 ifstate reads, 0 async ifd msgs, 0 async rtb msgs,  0 async bd msgs, 0 async mesh group msgs, 651 usec
Mar 25 10:12:53 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for IDLE                : user 0.000 s, sys 0.000 s, wall 5.005 s
Mar 25 10:12:53 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is CONFIG_RETRY
Mar 25 10:12:57 2489 rtsock_io.c:1991 rtsock_send() ERROR : rtsock_send Sucessive failures Can't assign requested address for ADD_AND_GET giving UP !
Mar 25 10:12:57 2489 dcd_sock.c:1689 dcd_process_config_write() ERROR : DCD_CONFIG_WRITE_FAILED: IFA em0.0 (family=2) configuration write failed for an IFA ADD: Can't assign requested address
Mar 25 10:12:57 2489 usage.c:75 dcd_trace_times() RSVRD :     Phase Usage for CONFIG_RETRY        : user 0.001 s, sys 0.000 s, wall 4.005 s
Mar 25 10:12:57 2489 dcd.c:855 dcd_new_phase() RSVRD : New phase is IDLE>show log messages | match "DFWD_TRASHED_RED_ZONE"
Apr 5 11:30:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:30:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:30:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:30:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:31:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:31:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:31:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:31:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:32:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:32:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:32:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:32:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:33:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:33:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:33:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:33:38  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:34:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
Apr 5 11:34:08  MX480 dfwd[2552]: DFWD_TRASHED_RED_ZONE: Trashed red zone pointer in block 0x3b3c000
.......
.............

Hello,

could you please help me to solve one design question.
Imagine, we have L3-switch (say qfx5110), with two routing-instances A and B configured on it.
Routing-instance A has bgp-peering, where it receives X.X.X.N/32 ipv4 prefixes from neighbor.
Then these /32 prefixes being aggregated to X.X.X.0/24 prefix.
Thus, we have in table A.inet.0 an aggregate route to X.X.X.0/24 with next-hop Reject (or Discard), when as least one X.X.X.N/32 prefix received from bgp-peer

Now we want to have this aggregated prefix X.X.X.0/24 in routing-instance B (B.inet.0), to be able to propagate it further.
We may achieve this by using for example instance-import command under routing-options statement.
But when aggregated prefix being imported to RI B, we receive it there with next-hop Reject (or Discard), same as it was originated in RI A.

The question is: can we change somehow the behavior, described above, to receive into B.inet.0 the prefix X.X.X.0/24 with next-hop = next-table A.inet.0, instead of Reject (or Discard)?

I've tried to go with:

set routing-instance B routing-options instance-import to-B

set policy-options policy-statement to-B term 1 from instance A
set policy-options policy-statement to-B term 1 from protocol aggregate
set policy-options policy-statement to-B term 1 then next-hop next-table A.inet.0
set policy-options policy-statement to-B term 1 then accept

but the prefix X.X.X.0/24 imported into B.inet.0 yet has Reject (Discard) as a next-hop.

The only workaround I were able to find (to have connectivity with announced /32 prefixes from RI B) is to import also all /32 prefixes into B.inet.0 from A.inet.0.
Now it works for me, however I dislike the scalability of such solution, as number of such /32 prefixes multiplies to number of routing-instances within the switch.

Thanks for your thoughts in advance.

Regards,
Tima K

You may use "generate" route instead of "aggregate" route to create /24 network. Generated route uses next-hop associated with the primary contributing route( in this case next hop of /32 route)  Please check this video for more details: 

https://www.youtube.com/watch?v=El8risTrdKk

MPC cards only support inline static NAT, only 1:1 source, destination or twice NAT is supported. For doing inline NAT, serivce-inline (si) interface must be used. There are two ways of configuring NAT: using interface or next-hop service-sets. The si- interface serves as a “virtual service PIC”.

There is a very good explanation of NAT in the book Juniper MX Series by Douglas Richard Hanks Jr., Harry Reynolds.

This article helped me as well.

https://www.juniper.net/documentation/en_US/junos/topics/concept/nat-inline-nat-overview.html

Regards

Hi Arix,

Thanks for the kudos! Can I kindly and respectfully ask you If this solves your problem? If yes please mark the previous post as "Accepted Solution" so we can help others too \/ 

Regards,

hi all,

Can anyone give more explaions about what does "0.0.0.0/0 exact" means? Any examples?

Thx,

A

exact means that it is exactly this route , in your case 0.0.0.0/0 and not any other routes included in this route-prefix

Thanks for the answer....

Can you give some examples for me... Is it possible?

Hi

The following example may help

you can get more informatin in the following link

https://www.dummies.com/programming/networking/juniper/understand-match-types-in-routing-filters/

Hope this helps.

Thanks for explaination and examples...

When looking at the acl in other vendor,  0.0.0.0/0 means "any" isn't it?

Hi all,

In multiple upstream ISP bgp configuration, how two ways dns traffic(lookup/recursive) should be allowed?

Yes that is correct. 0.0.0.0/0 means any route with any mask.

Hi Arix,

Exact knob means match only the specific route and nothing else

In your policy if you have "0.0.0.0/0 exact" then it will match default route (0.0.0.0/0) and nothing else

For this term to match you should have a route in "show route 0.0.0.0/0 exact" output

Without "exact" knob 0.0.0.0/0 will match each and every route in the routing table

Hope that make things clear.

PS: Please accept my response as solution if it answers your query, kuods are appreciated too!

Thanks

Vishal

Hi!
We have a softaware router on Linux (Quagga).
We bought Juniper MX204 and we need move configuration from Quagga to Juniper.

This is my first expiriense with Juniper and I don’t know much yet.

I have configuration on Quagga:

network X.X.X.X/20 route-map SETCOM
network Z.Z.Z.Z/22 route-map SETCOM

route-map SETCOM permit 10
set community YYYYY:0 YYYYY:10 YYYYY:100 YYYYY:150

but I can't find equivalent configuration for Juniper.

Thank you for your help and tips.

P.S. English isn't native to me, sorry for the mistakes

Hi all,

Kindly im try simulate lab based on this url https://www.juniper.net/documentation/en_US/release-independent/nce/topics/example/nce-178-collapsed-spine-optional-example.html .

Based on the url i'm able make from DC2 can communicate with DC1 and vice versa. But when i try for DHCP request from DC2 and DHCP Server at DC1 then it cannot get dhcp ip.

Is it have something miss in that url doc?

Thanks and appreciate any feedback.

Hello,

the equivalent on Junos would be this:

policy-options {
     prefix-list PREFIX1 {
        X.X.X.X/20;
	Z.Z.Z.Z/22;
    }
     policy-statement SETCOM {
        term SETCOM1 {
            from {
                prefix-list PREFIX1;
            }
	    then {
                community add COMMUNITY1;
		community add COMMUNITY2;
		community add COMMUNITY3;
		community add COMMUNITY4;
                accept;
            }
        }
    }
	community COMMUNITY1 members YYYYY:0;
	community COMMUNITY2 members YYYYY:10;
	community COMMUNITY3 members YYYYY:100;
	community COMMUNITY4 members YYYYY:150;
}